Xinference Documentation
Xinference is a cloud platform for deploying and serving open-source AI models at scale. With a single API call you can launch large language models (LLMs) and embedding models on dedicated GPU instances — billed per second, no infrastructure management required.
What is Xinference?
Xinference provisions GPU-backed inference clusters on demand. When you create a deployment, the platform:
- Selects the optimal GPU instance type for your chosen model
- Provisions the EC2 cluster in the cloud
- Downloads the model weights (or restores them from cache)
- Starts the Xinference inference server
- Returns an OpenAI-compatible endpoint you can query immediately
When you terminate a deployment the cluster is torn down and billing stops.
Core Concepts
| Concept | Description |
|---|---|
| Deployable Model | A model variant pre-approved for deployment (e.g. qwen2.5-instruct in int4 quantization on a specific GPU family). |
| Deployment | A running instance of a deployable model assigned to your account, with its own endpoint URL. |
| Cluster | The set of EC2 instances (supervisor + workers) backing a deployment. |
| Organization | A group of users sharing billing and deployment quotas. |
| Billing Account | Tracks usage and balance for an organization. Metered in GPU instance-seconds. |
Key Features
- OpenAI-compatible API — drop-in replacement for
openai.ChatCompletionandopenai.Embedding - Per-second billing — pay only for the time your model is running
- Model caching — popular models cached on S3 for fast cold-start times
- Multi-user organizations — invite teammates and share a single billing account
- SSO support — sign in with Google
- Idle auto-termination — clusters shut down automatically after a configurable idle period
Supported Model Types
- LLM (Chat) — instruction-tuned language models for chat and text generation
- Embedding — text embedding models for semantic search and RAG
Quick Links
Quickstart →
Launch your first model and query an OpenAI-compatible endpoint in minutes.
Supported Models →Browse the catalog of deployable LLMs and embedding models.
API Reference →Explore endpoints for deployments, models, organizations, and billing.
Self-Hosting →Run Xinference on your own infrastructure with Docker or AWS.