TGI, TabbyAPI, Aphrodite — alternate inference servers

After Ollama and vLLM / SGLang, there's a tier of inference servers that earn the slot for specific workloads. Each has a niche; none is the everywhere-default. For the bigger picture see the overview, hardware tiers, and quantization.

Text Generation Inference (TGI)

★ TGI (github.com/huggingface/text-generation-inference) — Hugging Face's production inference server.

• Apache 2.0 (license terms changed in 2023, then back to Apache 2.0 in 2024 after community pushback).
• Production-grade, well-tested, integrated with the Hugging Face ecosystem.
• Continuous batching, FlashAttention, tensor parallel, paged-attention-style memory.
• Supports FP16, BF16, FP8, AWQ, GPTQ, EETQ, Marlin.
• OpenAI-compatible endpoints + the older HF-style endpoints.
• Lost mindshare to vLLM in 2024–25. Still solid; less innovative now.

Use TGI if: you're already deep in the Hugging Face ecosystem (Spaces, Inference Endpoints, AutoTrain) and want the closest-to-HF-managed match for self-host. Otherwise, vLLM is more active.

TabbyAPI / exllamav2

★ TabbyAPI (github.com/theroyallab/tabbyAPI, AGPL-3.0) — OpenAI-compatible API server built on exllamav2, the EXL2 format's reference engine.

• ★ EXL2 quantization is the sweet spot for "high quality at low VRAM" — see quantization.
• Single-GPU focused; very fast on Ampere / Ada.
• Lower VRAM use than equivalent GPTQ / AWQ at similar quality.
• Roleplay / creative writing community is heavy here — long-context fiction generation has migrated to TabbyAPI + EXL2 because the quants compress better than GGUF Q4 at equivalent perceived quality.
• Lacks vLLM's batching / multi-tenant story.

Use TabbyAPI if: you're squeezing a 70B-class model into 24GB on a single 3090 / 4090 / 5090 and quality matters; or you're in the local-roleplay / creative-writing community where EXL2 is standard.

Aphrodite Engine

Aphrodite Engine (github.com/PygmalionAI/aphrodite-engine, AGPL-3.0) — fork of vLLM with extra sampling features and roleplay-community focus.

• Inherits vLLM's PagedAttention.
• Adds many sampling features (CFG, dynamic temperature, smoothing, mirostat, etc.) the roleplay community wants.
• Supports more quantization formats than upstream vLLM (GGUF, EXL2 in addition to AWQ / GPTQ).
• Production multi-tenant reliability is decent but less battle-tested than upstream vLLM.

Use Aphrodite if: you want vLLM throughput plus the sampler smorgasbord, especially for character-AI / SillyTavern-style workloads.

NVIDIA TensorRT-LLM

TensorRT-LLM (github.com/NVIDIA/TensorRT-LLM, Apache 2.0) — NVIDIA's optimised inference framework. AOT-compiles model graphs to TensorRT engines.

• Top-end raw throughput on NVIDIA hardware.
• Painful to use compared to vLLM — engine builds are slow, model support trails, every architecture change requires recompilation.
• Worth it if you're production-serving on H100/H200 and squeezing every percentage of throughput. Otherwise vLLM gets you 90% with 10% of the operational pain.

NVIDIA Triton Inference Server

Triton (github.com/triton-inference-server/server, BSD-3) — NVIDIA's general-purpose model server. Often used as a wrapper around vLLM or TensorRT-LLM for multi-model fleet management.

Use Triton if: you serve many models of different shapes (LLM + vision + traditional ML) and want one fleet manager. Otherwise it's overkill.

Ray Serve / KServe

For Kubernetes-native production deployments. Ray Serve + vLLM is a popular pairing for autoscaling LLM endpoints. KServe (formerly KFServing) is the Kubeflow-native option. Both are platform plumbing rather than inference engines themselves.

LocalAI

LocalAI (github.com/mudler/LocalAI, MIT) — drop-in OpenAI-compatible API supporting llama.cpp, transformers, Whisper, image gen, TTS, all in one server. The "everything-in-one" alternative to running Ollama + ComfyUI + Whisper separately.

Use LocalAI if: you want one container to serve text + image + audio. Less polished per-modality than the specialist tools but lower operational surface.

Pick this if…

• Single-GPU squeezing 70B into 24GB: ★ TabbyAPI + EXL2.
• Roleplay / creative writing community workflows: TabbyAPI or Aphrodite Engine.
• Hugging Face shop, want the matching self-host: TGI.
• Maximum NVIDIA throughput, willing to pay operational cost: TensorRT-LLM.
• Kubernetes multi-model fleet: Triton or Ray Serve + vLLM.
• One container for text + image + audio + TTS: LocalAI.
• Default production: vLLM. Default personal: Ollama.