Tooling
Self-Host

Ollama Deep Dive

The default starting-point inference engine for local LLMs in 2026 — install, models, GPU, OpenAI-compat API.

★ ★ Ollama is the default entry-point for self-hosted AI in 2026. A single binary, a single command (ollama run llama3.2), and you're talking to a local model. Under the hood it's a thin friendly wrapper around llama.cpp, with a model registry, GGUF management, automatic GPU detection, and an OpenAI-compatible API. If you're starting from the overview and the hardware tier guide, Ollama is almost certainly your first install.

Cross-links: pair with Open WebUI for a chat UI; with Aider or Continue.dev for coding; with LiteLLM as a routing proxy. For the underlying runtime see llama.cpp; for high-throughput serving graduate to vLLM / SGLang.

What Ollama actually is

  • ★ ★ A single Go binary that runs as a background service. Ships for macOS, Linux, Windows.
  • ★ ★ Model registry at ollama.com/library — ollama pull qwen3:32b downloads a model.
  • ★ GGUF + llama.cpp under the hood — every model is a quantized GGUF; the engine is upstream llama.cpp.
  • ★ OpenAI-compatible API on localhost:11434 — drop-in replacement for the OpenAI SDK by setting base_url.
  • ★ Modelfiles — Dockerfile-shaped recipes for setting system prompts, temperatures, context length, custom adapters.
  • ★ Auto GPU detection — NVIDIA CUDA, AMD ROCm, Apple Metal, Intel oneAPI; falls back to CPU.

License: MIT for the runtime; models carry their own licenses.

Quick start

# macOS (homebrew) or download from ollama.com
brew install ollama
 
# Linux
curl -fsSL https://ollama.com/install.sh | sh
 
# Run a model — auto-pulls on first run
ollama run llama3.2
ollama run qwen2.5-coder:32b
ollama run deepseek-r1:14b
 
# Background service
ollama serve   # listens on :11434
 
# OpenAI-compat client
curl localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.2", "messages": [{"role":"user","content":"hi"}]}'

The model library (May 2026 picks)

For the full landscape see ai-selfhost-models-overview. Highlights from ollama.com/library:

  • ★ ★ llama3.2 (Meta, 1B / 3B) — small, fast, surprisingly good at chat.
  • ★ ★ llama4 (Meta, multiple sizes; April 2025) — frontier-class open weights.
  • ★ ★ qwen3 (Alibaba, 0.5B–72B + MoE) — best-in-class multilingual; excellent at coding.
  • ★ ★ qwen2.5-coder (32B) — the local coding champion; pair with Aider or Continue.dev.
  • ★ ★ deepseek-r1 (1.5B–70B distillations of DeepSeek R1) — strong reasoning at every size.
  • ★ ★ deepseek-v3 — general-purpose; full version is huge but distillations exist.
  • ★ phi4 (Microsoft, 14B) — punches well above its weight on reasoning.
  • ★ gemma3 (Google, 1B–27B) — efficient; good at long context.
  • ★ mistral, mixtral, mistral-small — Mistral family.
  • ★ command-r-plus — Cohere; tool-use specialist.
  • ★ nomic-embed-text, mxbai-embed-large, bge-m3 — embeddings — see embeddings.
  • ★ llava, qwen2.5-vl — vision-language — see vision models.

Tag naming

ollama run qwen3:32b-instruct-q4_K_M

  • Model name — qwen3
  • Size tag — :32b (or :8b, :70b)
  • Variant — instruct, chat, base, coder
  • Quant — q4_K_M (default sweet spot), q5_K_M, q8_0, fp16. See quantization deep dive.

ollama run llama3.2 resolves to a sensible default tag (q4_K_M). Pin tags in production.

Modelfiles

FROM qwen2.5-coder:32b
 
PARAMETER temperature 0.2
PARAMETER num_ctx 32768
 
SYSTEM """
You are a concise senior engineer. Always show working code.
"""
ollama create my-coder -f Modelfile
ollama run my-coder

Use Modelfiles for: per-project system prompts, larger context windows than default, lower temperatures for code, attached LoRA adapters.

GPU configuration

  • NVIDIA — install latest CUDA; Ollama auto-detects. Set OLLAMA_FLASH_ATTENTION=1 for FlashAttention on Ampere+ (3060 and newer).
  • AMD ROCm — ROCm 6.x ships with usable Ollama support by mid-2025; older RDNA1/2 cards may need HSA_OVERRIDE_GFX_VERSION. Newer 7900 XTX works out of the box.
  • Apple Silicon — Metal works automatically; no setup. Set OLLAMA_NUM_PARALLEL=1 for max single-request speed.
  • Intel Arc — basic support via SYCL / Vulkan; experimental.

Important env vars:

  • OLLAMA_HOST=0.0.0.0:11434 — bind to LAN (default is localhost only).
  • OLLAMA_MODELS=/mnt/big-disk/ollama — change the model storage directory.
  • OLLAMA_KEEP_ALIVE=30m — how long to keep models in VRAM after idle.
  • OLLAMA_NUM_PARALLEL=4 — parallel requests per model.
  • OLLAMA_MAX_LOADED_MODELS=2 — number of models loaded simultaneously.

OpenAI-compatible API

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
r = client.chat.completions.create(
    model="qwen3:32b",
    messages=[{"role": "user", "content": "Refactor this function..."}],
)

This works with any OpenAI SDK or library that takes a base_url. Aider, Continue.dev, Cline, the Vercel AI SDK, LangChain, LlamaIndex — all work.

Tool use / function calling

Ollama supports OpenAI-style tool calling for models that have tool-use chat templates (Llama 3.1+, Qwen 2.5+, Llama 4, DeepSeek V3, Mistral with tool-use). Quality is model-dependent — Qwen 2.5 / 3 and Llama 4 are the most reliable for tool calls in 2026.

Structured output

format: "json" mode forces JSON output. New in 2024–25, structured output via JSON Schema is supported on most models that include grammar-constrained decoding via llama.cpp.

Common honest gotchas

  • Default context is short. Most Ollama tags ship at 4K or 8K context. To use a model's full context, set num_ctx in the Modelfile or pass options.num_ctx per request. Bigger context costs VRAM.
  • One concurrent request unless you bump OLLAMA_NUM_PARALLEL. Default is 1 for single-user assumption.
  • Quants are pre-baked. You can't switch between quants on the fly; pull the specific tag you want.
  • No batching like vLLM. For multi-tenant or production throughput, graduate to vLLM / SGLang.
  • Model storage gets big fast. A casual user accumulates 100GB of models in a month.
  • Llama 4 / Qwen 3 large variants are MoE — they fit in less compute than dense equivalents but still need the full weights in RAM.

When to graduate from Ollama

Pick this if…

  • You're starting: Ollama, no contest. ★ ★
  • You want a simple chat UI on top: Ollama + Open WebUI.
  • You want a coding assistant on top: Ollama + Aider or Continue.dev.
  • You're on Mac and want GUI install: Ollama still works fine; LM Studio is the GUI alternative.
  • You're going to production: keep Ollama for dev; deploy with vLLM.

Previous

Hardware Tier Guide

Next

llama.cpp

On this page