Chat Completions

The /v1/chat/completions endpoint generates text responses from instruction-tuned LLMs.

Endpoint

POST /v1/chat/completions

Request

Required Fields

Field	Type	Description
`model`	string	Model name (e.g. `qwen2.5-instruct`)
`messages`	array	Conversation history

Messages Format

[
  {"role": "system", "content": "You are a helpful assistant."},
  {"role": "user", "content": "What is Xinference?"},
  {"role": "assistant", "content": "Xinference is a cloud inference platform..."},
  {"role": "user", "content": "How do I get started?"}
]

Valid roles: system, user, assistant

Optional Fields

Field	Type	Default	Description
`temperature`	float	1.0	Sampling temperature (0–2). Lower = more deterministic
`top_p`	float	1.0	Nucleus sampling probability
`max_tokens`	integer	null	Maximum tokens in the response
`stream`	boolean	false	Stream tokens via SSE
`stop`	string or array	null	Stop sequences
`n`	integer	1	Number of completions to generate
`presence_penalty`	float	0	Penalize tokens already in the context
`frequency_penalty`	float	0	Penalize frequently occurring tokens

Basic Example

The platform authenticates with the session cookie set at sign-in (see Authentication). The cURL example below sends that cookie. The Python snippet illustrates the OpenAI-compatible request shape that the endpoint accepts.

cURL:

curl https://api.xinference.co/v1/chat/completions \
  --cookie "session=<your-session-cookie>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-instruct",
    "messages": [
      {"role": "system", "content": "You are a concise technical assistant."},
      {"role": "user", "content": "Explain vector embeddings in 2 sentences."}
    ],
    "temperature": 0.3,
    "max_tokens": 128
  }'

Python:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.xinference.co/v1",
    api_key="...",  # platform auth uses the session cookie
)

response = client.chat.completions.create(
    model="qwen2.5-instruct",
    messages=[
        {"role": "system", "content": "You are a concise technical assistant."},
        {"role": "user", "content": "Explain vector embeddings in 2 sentences."},
    ],
    temperature=0.3,
    max_tokens=128,
)

print(response.choices[0].message.content)

Streaming

For real-time token streaming, set stream=True:

stream = client.chat.completions.create(
    model="qwen2.5-instruct",
    messages=[{"role": "user", "content": "Write a haiku about AI."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)

Multi-Turn Conversations

Maintain conversation history by appending each assistant reply to the messages array:

messages = [{"role": "system", "content": "You are a helpful assistant."}]

def chat(user_input: str) -> str:
    messages.append({"role": "user", "content": user_input})
    response = client.chat.completions.create(
        model="qwen2.5-instruct",
        messages=messages,
    )
    reply = response.choices[0].message.content
    messages.append({"role": "assistant", "content": reply})
    return reply

print(chat("What is 2 + 2?"))
print(chat("Now multiply that by 10."))

Usage Tracking

Each response includes token usage:

print(response.usage.prompt_tokens)
print(response.usage.completion_tokens)
print(response.usage.total_tokens)

Token usage is the basis for billing. See Usage Metering →.