Self-Hosting Xinference

Xinference can be deployed in your own cloud account or on-premises. This gives you full control over data residency, network configuration, and billing.

Architecture

A self-hosted Xinference deployment consists of:

┌─────────────────────────────────────────┐
│  Your Infrastructure                     │
│                                          │
│  ┌──────────┐    ┌────────────────────┐ │
│  │ Frontend │◄──►│  Backend (FastAPI)  │ │
│  │ (React)  │    │  + PostgreSQL DB    │ │
│  └──────────┘    └────────┬───────────┘ │
│                            │             │
│                   ┌────────▼───────────┐ │
│                   │  AWS EC2 Clusters  │ │
│                   │  (GPU Workers)     │ │
│                   └────────────────────┘ │
└─────────────────────────────────────────┘

Components:

Component	Description
Backend	Python FastAPI application — handles auth, deployments, billing, and proxies inference
Frontend	React SPA — dashboard UI
Database	PostgreSQL (or SQLite for development)
AWS	EC2 instances for GPU inference workers

Prerequisites

AWS account with permissions to launch EC2 instances, create IAM roles, and manage security groups
Docker (optional, recommended for production)
Python 3.11+ and uv for local development
Node.js 18+ for frontend builds

Deployment Options

Docker Compose (Recommended)

The quickest way to run the full stack locally or on a single server. See Docker Setup →.

Render / PaaS

The backend can be deployed as a web service on Render, Railway, Fly.io, or similar platforms. The frontend can be deployed to Vercel, Netlify, or as a static site on S3/CloudFront.

Kubernetes

For production-scale deployments, the backend is stateless (except for the database) and can run as a Kubernetes Deployment with horizontal pod autoscaling.

Quick Start

git clone https://github.com/TeohCapital/xinference-saas
cd xinference-saas
cp backend/.env.example backend/.env
# Edit .env with your configuration
docker compose up