Deployments

A deployment is a running instance of an AI model on dedicated GPU infrastructure. Xinference manages the full lifecycle — from provisioning cloud resources to tearing them down when you are done.

Deployment Lifecycle

pending → provisioning → starting → running → terminating → terminated

Infrastructure

Each deployment runs on an AWS cluster consisting of:

• Supervisor node — coordinates the cluster, exposes the Xinference HTTP API
• Worker nodes — GPU instances that run inference

The number and type of worker nodes depends on the selected model. Xinference automatically picks the most cost-effective GPU type based on the model size and quantization.

Cluster Idle Auto-Termination

To prevent runaway costs, Xinference automatically terminates clusters that have been idle for too long. The default idle TTL is 15 minutes. A cluster is considered idle when no inference requests have been received for that duration.

You can configure the idle TTL for self-hosted deployments via XINFERENCE_SAAS_CLUSTER_IDLE_TTL_SECONDS.

Model Caching

For supported models, Xinference maintains an S3-backed model cache. When a cached model is deployed:

• Model weights are restored from S3 to the worker instance's local disk before starting
• Cold-start time drops from 5–15 minutes (fresh download) to 30–90 seconds (cache restore)

Cache availability is indicated on each model card in the dashboard.

Endpoints

Once a deployment reaches running, it exposes an OpenAI-compatible endpoint. You can find the endpoint URL on the deployment detail page, or retrieve it from the API response.

All inference endpoints are authenticated with your API key.

Quotas

Deployment quotas are configured per organization. Contact support if you need higher concurrency limits.

Next Steps

• Create a Deployment →
• Deployment Status →
• Manage Deployments →