AWS Setup

Xinference provisions EC2 GPU instances in your AWS account to run inference workloads. This guide walks through the required AWS configuration.

IAM Permissions

The backend service needs an IAM role or credentials with the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:RunInstances",
        "ec2:TerminateInstances",
        "ec2:DescribeInstances",
        "ec2:DescribeInstanceStatus",
        "ec2:CreateTags",
        "ec2:DescribeTags"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:PassRole"
      ],
      "Resource": "arn:aws:iam::*:instance-profile/xinference-worker-profile"
    }
  ]
}

Worker Instance Profile

EC2 worker instances need an instance profile that allows them to pull model weights from S3 (if using model caching):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-model-cache-bucket",
        "arn:aws:s3:::your-model-cache-bucket/*"
      ]
    }
  ]
}

Create an instance profile named xinference-worker-profile and attach this policy.

Security Groups

Backend Security Group

Outbound rules needed by the backend: - Port 9997 TCP → worker supervisor nodes (Xinference default port) - Port 443 HTTPS → AWS APIs

Worker Security Group

AMI (Amazon Machine Image)

Xinference requires a GPU-enabled AMI with: - NVIDIA CUDA drivers - Docker (with NVIDIA Container Toolkit) - The xprobe/xinference Docker image pre-pulled (optional, speeds up startup)

Building the AMI

A starting point using the AWS Deep Learning Base AMI:

1. Launch a GPU instance with the Deep Learning Base AMI (Amazon Linux 2)
2. Pull the Xinference image: docker pull xprobe/xinference:latest
3. Install cloud-init scripts for model cache sync
4. Create an AMI from the instance
5. Set XINFERENCE_SAAS_AWS_AMI_ID to the new AMI ID

VPC and Networking

For production deployments: - Place backend and workers in the same VPC to avoid cross-region data transfer costs - Use a private subnet for workers (set XINFERENCE_SAAS_SUPERVISOR_ENDPOINT_HOST=private) - Ensure the backend can reach worker private IPs (check security group rules and routing)

For local testing where the backend runs outside AWS:

XINFERENCE_SAAS_AWS_ASSOCIATE_PUBLIC_IP=true
XINFERENCE_SAAS_SUPERVISOR_ENDPOINT_HOST=public

Model Cache (S3)

To enable fast cold-starts, sync model weights to an S3 bucket:

# Example: cache Qwen2.5-7B-Instruct weights
aws s3 sync ~/.cache/huggingface/hub/models--Qwen--Qwen2.5-7B-Instruct \
  s3://your-model-cache-bucket/qwen2.5-7b-instruct/

Set the model_cache_s3_uri, model_cache_host_path, and model_path fields on the deployable model record to enable cache restore on worker startup.

Cost Estimation

Typical monthly costs for a lightly used deployment:

Use AWS Reserved Instances for GPU workers if you have predictable usage patterns.