vLLM and SGLang

★ ★ vLLM and ★ ★ SGLang are the two FOSS inference servers you graduate to when Ollama hits its multi-user / batching limits. PagedAttention (vLLM, 2023) and RadixAttention (SGLang) brought server-side throughput improvements that made on-prem GPT-3.5-class serving genuinely practical. If you have multiple users, agentic loops, or batch workloads, this is your tier.

For single-user chat, stay on Ollama. For the production big-picture see the overview and hardware tiers; for the dataset-format choices see quantization; for additional alternates see TGI / TabbyAPI.

vLLM

★ ★ vLLM (github.com/vllm-project/vllm, Apache 2.0) — the dominant FOSS production inference engine.

• ★ PagedAttention — KV-cache memory management that treats GPU memory like virtual memory, eliminating fragmentation and enabling much higher concurrent batch sizes.
• ★ Continuous batching — incoming requests join in-flight batches dynamically; no head-of-line blocking.
• ★ OpenAI-compatible API — drop-in.
• ★ Wide model support — Llama, Qwen, DeepSeek, Mistral, Mixtral, Phi, Gemma, Command-R, Llama 4, MoE models, vision models, and growing.
• ★ Quantization formats — FP16, BF16, FP8 (Hopper / Ada), AWQ, GPTQ, Marlin, INT8 — but not GGUF. See quantization.
• ★ Tensor parallel + pipeline parallel for multi-GPU / multi-node serving.
• ★ Speculative decoding — small draft model + big target.
• ★ LoRA serving — load multiple LoRA adapters at once and route per-request.
• ★ Prefix caching — long shared prefixes (system prompts, RAG context) get cached automatically.
• License: Apache 2.0.

pip install vllm
vllm serve Qwen/Qwen3-32B-Instruct \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92

The Python API is also fine for embedding into applications, but most production users run it as a server.

SGLang

★ ★ SGLang (github.com/sgl-project/sglang, Apache 2.0) — newer (2024) competitor / collaborator to vLLM.

• ★ RadixAttention — generalises prefix caching with a radix tree; particularly fast for many-turn chats and agentic workloads with shared context.
• ★ Structured output / constrained decoding — first-class support; faster than vLLM's grammar implementations.
• ★ Frontend DSL — sglang Python primitives for multi-step LLM programs (fork, gen, select) — not just a server but a programming model.
• ★ Strong on DeepSeek, Llama, Qwen — often first to support new big-model architectures (DeepSeek V3 multi-head latent attention support shipped quickly).
• ★ Lower latency than vLLM on some workloads, particularly with long shared prefixes.
• License: Apache 2.0.

pip install sglang[all]
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-32B-Instruct \
  --tp 2 --port 30000

vLLM vs. SGLang in May 2026

• vLLM is the broader-platform default. More models supported on day-1, more deployment-tooling integrations (Ray, Kubernetes, KServe), bigger community.
• SGLang is faster on workloads with heavy prefix sharing (long system prompts, RAG, multi-turn agents). Strong choice for DeepSeek and reasoning-heavy workloads.
• Both are Apache 2.0, both have OpenAI-compatible servers, both run on NVIDIA (CUDA) primarily; AMD ROCm support exists for both with caveats.
• Pick vLLM if you want the safer mainstream choice.
• Pick SGLang if your workload has long shared prefixes or you want structured output performance.
• Many production deployments run both — vLLM for the bulk of traffic, SGLang for specific high-prefix workloads.

When to use vLLM / SGLang vs. Ollama

Hardware reality

• vLLM and SGLang shine on GPUs with full FP16/BF16/FP8 weights, not heavily quantized GGUFs. You need real VRAM.
• 70B FP16 needs 140GB VRAM — 2× A100 80GB, 1× H100 80GB + tensor parallel, or AWQ/GPTQ-quantized to fit on 24GB cards.
• Tier 3+ (hardware guide) is the floor for serious vLLM / SGLang use.
• For Tier 1–2 hardware running quantized models for personal use, stay on Ollama.

Operational practicalities

• Run behind LiteLLM as a routing proxy if you want logging, budgets, multi-model fallback.
• Use Open WebUI as the chat front-end pointed at vLLM's OpenAI endpoint.
• Pin your model version explicitly. Production serving is not the place for :latest.
• Tune --max-model-len — set it to the max context you actually need; bigger means more KV cache RAM.
• Tune --gpu-memory-utilization — 0.92 is a sane default; lower if you see OOM, higher if you have headroom.

Pick this if…

• Multi-user serving (5+ concurrent): vLLM.
• Agentic loops with shared system prompts: SGLang for prefix caching wins.
• Production behind an API gateway: vLLM is the more mainstream choice.
• Personal single-user box: stay on Ollama.
• Hopper / Ada with FP8: vLLM with FP8 quants.
• Bleeding-edge model architectures (DeepSeek V3, Llama 4 MoE): SGLang is often first.
• Want to hand-build a multi-step LLM program: SGLang's frontend DSL.