Model Types

Xinference supports three categories of AI models, each with different API endpoints and use cases.

LLM (Large Language Models)

LLMs are instruction-tuned language models for chat and text generation tasks.

Use cases: - Conversational AI / chatbots - Document summarization - Code generation and review - Question answering over documents (RAG) - Data extraction and classification

API endpoint: /v1/chat/completions

Key parameters:

Example:

response = client.chat.completions.create(
    model="qwen2.5-instruct",
    messages=[{"role": "user", "content": "Explain transformers in one sentence."}],
    temperature=0.7,
    max_tokens=256,
)

Embedding Models

Embedding models convert text into dense vector representations for semantic similarity tasks.

Use cases: - Semantic search - Retrieval-Augmented Generation (RAG) - Document clustering - Duplicate detection

API endpoint: /v1/embeddings

Key parameters:

Example:

response = client.embeddings.create(
    model="bge-m3",
    input=["Xinference is a cloud inference platform.", "Deploy AI models easily."],
)

vectors = [item.embedding for item in response.data]
print(len(vectors[0]))  # e.g. 1024

Note: Embedding dimensions vary by model. Check the Supported Models table for each model's output dimension.