llama.cpp

★ ★ llama.cpp is the foundational FOSS inference engine for local LLMs. Started by Georgi Gerganov in March 2023 as a "what if I ran LLaMA on a MacBook" experiment, it became the substrate under Ollama, LM Studio, KoboldCpp, Jan, and most "I run LLMs locally" tools. The GGUF file format is its model format and now the universal standard for quantized open weights.

If you're running Ollama, you're running llama.cpp. Most users will never touch it directly — Ollama is the friendlier wrapper. Direct llama.cpp use is for power users who want maximum control. See the overview, hardware tiers, and quantization.

What llama.cpp is

• ★ ★ C++ inference of LLaMA-family models on CPU and GPU. Pure C++; no Python; no PyTorch; no CUDA toolkit required to build.
• ★ ★ GGUF format — single-file model with weights, tokenizer, metadata, and quantization parameters all bundled. The de-facto open-weight distribution format.
• ★ Backends: CUDA, Metal (Apple), ROCm (AMD), Vulkan, SYCL (Intel), CPU (AVX2 / AVX-512 / NEON).
• ★ Server mode with OpenAI-compatible HTTP API.
• ★ Active community. Hundreds of contributors; merges multiple PRs/day; among the most active OSS projects in AI.
• License: MIT.

GitHub: github.com/ggml-org/llama.cpp

When to use llama.cpp directly vs. Ollama

• Use Ollama if: you want it to "just work," you want a model registry, you want an API on a stable port.
• Use llama.cpp directly if:
  • You want to build with specific compiler flags / optimizations.
  • You want bleeding-edge features (new model architectures often land here first).
  • You want to convert / requantize / merge GGUFs yourself.
  • You're on an unusual platform (RISC-V, Termux, embedded).
  • You want minimal binary size for embedded / edge use.

Building from source (it's not that bad)

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
 
# CPU-only
cmake -B build && cmake --build build --config Release -j
 
# CUDA
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j
 
# Metal (Mac, default on)
cmake -B build && cmake --build build --config Release -j
 
# Vulkan (cross-vendor GPU)
cmake -B build -DGGML_VULKAN=ON && cmake --build build --config Release -j

CLI tools

• llama-cli — interactive REPL or one-shot completion.
• llama-server — HTTP server with OpenAI-compatible API.
• llama-bench — benchmark token generation rates per backend.
• llama-quantize — convert FP16 GGUF to Q4_K_M, Q5_K_M, Q6_K, Q8_0.
• llama-imatrix — compute importance matrices for high-quality quantization.
• llama-perplexity — measure quantization quality vs. base model.

Running a model

# One-shot
./llama-cli -m models/qwen3-32b-q4_K_M.gguf -p "Write a haiku" -n 128
 
# Server mode (OpenAI-compatible)
./llama-server -m models/qwen3-32b-q4_K_M.gguf -c 32768 --host 0.0.0.0 --port 8080
 
# Use with any OpenAI client by setting base_url=http://localhost:8080/v1

Key flags:

• -c N — context size in tokens.
• -ngl N — number of layers to offload to GPU (use a high number for full GPU offload).
• -t N — CPU threads.
• --flash-attn — enable FlashAttention (highly recommended on Ampere+).
• --n-predict N — max tokens to generate.
• --temp 0.7 --top-p 0.9 — sampling parameters.
• -fa — alias for FlashAttention; -cnv for chat mode; -hfr <repo> to download from Hugging Face.

Where to get GGUFs

• Hugging Face — search for GGUF quants. Look for bartowski, unsloth, TheBloke (legacy), lmstudio-community, and the official model author when they publish GGUFs (Meta, Alibaba, DeepSeek, Mistral all do for many recent releases).
• Ollama library — ollama itself stores GGUFs locally; you can extract them from ~/.ollama/models/.
• Convert yourself — clone the model from HF, run convert_hf_to_gguf.py from the llama.cpp repo to produce an FP16 GGUF, then quantize with llama-quantize.

GGUF, briefly

• Single file containing weights, tokenizer, prompt template, and metadata.
• Multiple quants per model are pre-built and distributed separately — Q4_K_M, Q5_K_M, Q6_K, Q8_0, F16. See quantization deep dive.
• Importance matrix (imatrix) quants — higher quality at the same bit-rate using calibration data; bartowski's "i1" quants are the popular ones.
• GGUF v3+ has split-file support for huge models (Llama 4 405B in 4 parts, etc.).

What llama.cpp supports that Ollama hides

• Speculative decoding — run a small "draft" model alongside a big "target" model for 2–3× throughput; supported via --draft-model.
• Custom RoPE / context scaling — --rope-scaling yarn --rope-freq-scale for stretching context length on models that support it.
• CFG (classifier-free guidance) — --cfg-scale for steering generation.
• Grammar-constrained sampling — pass GBNF grammars to enforce JSON / specific output structure.
• Logits bias / token bans — fine-grained control of sampling.
• Tensor parallel split across multiple GPUs (--tensor-split).

llama.cpp on every platform that matters

• Apple Silicon — Metal backend; ★ extremely well-optimized; the original target.
• NVIDIA CUDA — fastest path on Linux/Windows.
• AMD ROCm — works on RDNA2+; HIP under the hood.
• Vulkan — cross-vendor GPU (NVIDIA / AMD / Intel / mobile); slower than native CUDA / ROCm but extremely portable.
• Termux on Android — yes, you can run a 3B model on a phone.
• Raspberry Pi — yes, and it's slow but works.
• WebAssembly — llama.cpp compiles to WASM; basis of Transformers.js' GGUF support.

Honest tradeoffs vs. vLLM / TGI

• llama.cpp wins at: single-user inference, small VRAM footprints, Apple Silicon, exotic hardware, simple deployment.
• vLLM / SGLang win at: multi-user batched throughput, full-precision serving, production at scale. See vLLM / SGLang deep dive.

Pick this if…

• You're an end-user: don't use llama.cpp directly; use Ollama.
• You want maximum control on a single machine: llama.cpp.
• You're building a product around it: llama.cpp's library API (libllama) embeds cleanly.
• You're on weird hardware: llama.cpp is your only option.
• You're benchmarking quantization: llama-perplexity is the standard tool.
• You're doing speculative decoding / advanced sampling: llama.cpp directly.