Tooling
Self-Host

Self-Hosted AI Overview

Run LLMs on your own hardware in 2026 — the Llama 4, Qwen 3, DeepSeek V3 / R1, Mistral, Phi-4 era.

Self-hosting AI in May 2026 is genuinely different from 2023–24. Llama 4 (Meta, April 2025) shipped open weights at GPT-4-class quality. Qwen 3 (Alibaba) and DeepSeek V3 / R1 match or beat closed models on coding and reasoning. Mistral, Phi-4 (Microsoft), Gemma 3 (Google) and Command-R+ round out the open landscape. Inference engines like Ollama, vLLM, and llama.cpp are mature. The hardware you need has dropped dramatically — a used $700 GPU runs a model that beats GPT-3.5 on most tasks, and a $2,000 build runs models that rival GPT-4-of-2023. This is the deep-dive section.

This is the landing for a deep self-hosted AI cluster. Start with the hardware tier guide — the headline page — and then pick an inference engine, a chat UI, and a workflow. For the quick TypeScript-app angle see ai-llm; for the broader productivity angle see prod-ai-productivity-assistants; the older shallow-survey ops page is at selfhost-ai-llm.

Why self-host AI in 2026

  • ★ ★ Privacy. Nothing leaves your machine. Health, financial, legal, journal entries, customer data, source code — none of it gets logged by an API provider, none of it gets used for training. This is the dominant reason most self-hosters do this.
  • ★ ★ No rate-limits, no token meter. Run agents in tight loops. Throw 200K-token contexts at a 70B model. Re-summarize the same document 50 ways. The marginal cost is electricity.
  • ★ Cost at heavy use. A used RTX 3090 ($800) running 24/7 costs less per year than a $20/mo ChatGPT Plus subscription if you actually saturate it. Break-even depends on usage — see ai-selfhost-cost-cloud-vs-local.
  • ★ Control. Pick your model. Pin its version. Modify the system prompt globally. Add tools. Turn off "safety" lectures for a coding agent. Fine-tune on your own writing.
  • ★ Offline capable. The model still works on a plane, a sailboat, a power-cut, a ranch with a starlink that flakes. Local inference is genuinely resilient.
  • Hobby / learning. This is the ML / systems hobby of the decade. Quantization, batching, KV-cache, speculative decoding, FlashAttention — you learn a lot building this.

Honest tradeoffs vs. paid APIs

  • Frontier capability gap. GPT-5, Claude Opus 4.7, Gemini 2.5 Ultra (May 2026) are still meaningfully better than the best open weights at the very hardest reasoning tasks. The gap has narrowed to roughly "open is 6–12 months behind on the frontier" — better than 2024's 18–24 months — but it persists.
  • Speed. A frontier API gives you 100+ tokens/sec at any context size. A 70B model on a 3090 gives 8–15 tok/s. Acceptable for chat; punishing for "agent that calls the LLM 200 times for one task."
  • Setup time. Paid API: sign up, paste a key. Self-host: pick hardware, pick driver versions, pick a runtime, pick a model, pick a quant, pick a UI, debug GPU passthrough, debug context length, debug RAG. The Ollama path has compressed this dramatically but it isn't zero.
  • Hardware cost up front. $700–$3,500 to do this seriously. Real money.

For honest math on this break-even, see ai-selfhost-cost-cloud-vs-local.

The mental model

Pick four things, in order:

  1. Hardware — what you have or what you'll buy. See ai-selfhost-hardware-tiers.
  2. Inference engine — the runtime that loads the model and serves requests. Most people: Ollama. Power users: llama.cpp, vLLM / SGLang, or LM Studio.
  3. Model — the weights. See ai-selfhost-models-overview, coding models, vision / multimodal, embeddings.
  4. Interface / workflow — the UI you talk to. Chat: Open WebUI, LibreChat / AnythingLLM, Big-AGI / Msty. Coding: Aider, Cline / Roo Code, Continue.dev, OpenHands / SWE-agent. RAG: ai-selfhost-rag-local. Voice: voice stack. Image: ComfyUI.

The May-2026 landscape, in one page

  • Inference engines (FOSS): ★ ★ Ollama, ★ ★ llama.cpp, ★ ★ vLLM, ★ SGLang, ★ TabbyAPI / exllamav2, ★ Aphrodite Engine, ★ Jan, ★ GPT4All. Closed-but-free: LM Studio.
  • Frontier open models: Llama 4 (Meta, multiple sizes including a 400B+ MoE), Qwen 3 (Alibaba, dense + MoE), DeepSeek V3 (general) + R1 (reasoning), Mistral Large 2 / Mixtral, Phi-4, Gemma 3, Command-R+ 2025.
  • Coding-specialist models: Qwen2.5-Coder 32B (best small coder), DeepSeek-Coder-V2, CodeLlama, Codestral, StarCoder 2.
  • Chat UIs (FOSS): ★ ★ Open WebUI, ★ ★ LibreChat, ★ AnythingLLM, ★ Lobe Chat, ★ Big-AGI, Msty.
  • Agentic coding (FOSS): ★ ★ Aider, ★ ★ Cline, ★ ★ Continue.dev, ★ Roo Code, ★ OpenHands, ★ SWE-agent, ★ Plandex.
  • Image gen (FOSS): ★ ★ ComfyUI, ★ AUTOMATIC1111, ★ InvokeAI, ★ Fooocus.
  • Voice (FOSS): ★ ★ Whisper / WhisperX, ★ Piper TTS, ★ Coqui XTTS, ★ Kokoro TTS.
  • Observability (FOSS): ★ ★ LiteLLM proxy, ★ ★ Langfuse self-host, ★ Helicone self-host.

What's still hard / unsolved

  • 70B at full quality on a single consumer GPU. Q4 fits in 24GB, full FP16 doesn't. Q4 is fine for chat; some coding / long-context tasks notice the loss.
  • Fast prompt processing on Apple Silicon. Generation is fine; the long-context "thinking" pause is rough.
  • Truly autonomous coding agents. OpenHands / SWE-agent are impressive demos but unreliable in production with local 30B–70B. Aider with a strong model + human-in-loop is the local gold standard.
  • Voice latency end-to-end. STT + LLM + TTS + speaker round-trip in <500ms is hard; doable on a 3090 with small models.
  • Multimodal at home. Qwen2.5-VL is good; the gap to GPT-4o-class is bigger than the text-only gap.

Pick this if…

Previous

Self-Host — AI, Ops, Security & Useful Self-Hosted

Next

Hardware Tier Guide

On this page