Quantization — GGUF, AWQ, GPTQ, EXL2, FP8

Quantization shrinks model weights from FP16 (16 bits/weight) to 8, 6, 5, 4, or even 2–3 bits per weight, trading a small amount of quality for a large amount of VRAM and speed. Q4_K_M (GGUF, ~4.5 bits/weight effective) is the universal sweet spot — most users should use it without thinking. This page is the deeper map for when you need to think.

Cross-links: Ollama ships GGUF; llama.cpp is the reference; vLLM / SGLang prefer AWQ / GPTQ / FP8 for serving; TabbyAPI is the EXL2 home; hardware tiers for VRAM math.

Quick map

GGUF — the community standard

The format pioneered by llama.cpp. Single-file, self-contained, runs on every CPU/GPU/Mac.

Quant types in GGUF

• Q2_K — extreme compression; often unusable except for biggest models.
• Q3_K_S / M / L — small models suffer; usable on 70B+.
• Q4_K_S — small variant.
• ★ ★ Q4_K_M — the sweet spot. ~4.5 bpw; small quality drop vs FP16; small memory; runs everywhere.
• Q5_K_S / M — slightly better quality, moderately more VRAM.
• ★ Q6_K — near-FP16 quality; ~6.5 bpw; for when you have the VRAM.
• Q8_0 — basically FP16 quality at half the size; what you want if VRAM permits.
• F16 / BF16 — full precision; reference.

Imatrix (importance-matrix) quants

Calibration-aware quantization using llama.cpp's imatrix tool. Quants computed with imatrix preserve quality better at the same bit-rate — particularly noticeable at Q3 and below. Bartowski's "i1" quants on Hugging Face are the popular ones; use them when available.

AWQ — Activation-aware Weight Quantization

• Designed for production inference.
• 4-bit weights; activations stay FP16/BF16.
• Strong quality at 4 bits.
• vLLM / TGI / SGLang first-class.
• Per-model: search HF for TheBloke or casperhansen/...-AWQ or model authors' AWQ releases.

GPTQ

• Older 4-bit weight-only quantization.
• Works in vLLM, exllamav2, TabbyAPI, transformers (slow).
• Largely superseded by AWQ for new releases but still widely available.

EXL2

• Mixed-bit-rate quantization (different layers at different bit widths) for best-quality-per-VRAM.
• exllamav2 / TabbyAPI only.
• Bit-rate measured in bpw (bits-per-weight): 4.0bpw, 5.0bpw, 6.0bpw, 8.0bpw.
• 6.0bpw is roughly Q6_K quality; 5.0bpw is roughly Q4_K_M / Q5_K_M.
• ★ Often the best quality fit for "70B in 24GB" use cases.

FP8 — production-grade quantization on H100 / Ada

• Two flavours: e4m3 (more dynamic range) and e5m2 (more precision).
• Hardware-accelerated on Hopper / Ada Lovelace (H100, H200, RTX 4090, RTX 5090).
• vLLM ships FP8 support; quality is excellent — nearly indistinguishable from BF16.
• Production teams running on H100 frequently default here.

Marlin

• High-throughput INT4 inference kernel for vLLM on Ampere+ NVIDIA.
• Used under the hood when serving GPTQ quants on supported hardware.

When to pick which

• Personal use, Ollama / LM Studio / single-user: ★ ★ GGUF Q4_K_M (or Q6_K if you have the VRAM).
• Production multi-user serving: AWQ on vLLM, or FP8 if on Hopper/Ada.
• Fitting a 70B into 24GB on a single 3090/4090/5090: EXL2 at 4.5–5.0bpw via TabbyAPI.
• Roleplay / creative writing: GGUF Q4–Q6 or EXL2 — both have communities; sampler matters more.
• Maximum quality, no compromise: BF16 / FP16 if you have the VRAM (rare at home).

How much VRAM does each take?

For a 70B model:

• FP16 / BF16: ~140GB
• Q8_0: ~75GB
• Q6_K: ~58GB
• Q5_K_M: ~50GB
• ★ Q4_K_M: ~42GB
• AWQ 4-bit: ~38GB
• 4.5bpw EXL2: ~40GB

Add ~20% for KV cache and runtime overhead, especially with longer contexts.

Quality drop perception

• Q8 → Q6_K: essentially imperceptible.
• Q6_K → Q4_K_M: small; noticeable on hardest tasks at smaller model sizes (sub-13B).
• Q4_K_M → Q3_K_M: noticeable; only worth it on biggest models where you can't fit Q4.
• Q3 → Q2: large; rarely worth it.

Bigger models take quantization better than smaller ones — a 70B-Q4 is closer to 70B-FP16 than a 7B-Q4 is to 7B-FP16.

Honest framing

• Q4_K_M is the right answer for ~95% of users. Don't overthink it.
• Imatrix quants are a free quality bump when available — prefer them.
• The roleplay / SillyTavern community has measurable preferences for EXL2 over GGUF at the same VRAM tier; if quality matters and you're on a single 24GB GPU, TabbyAPI + EXL2 is worth the operational complexity.
• Production serving is a different game — AWQ + vLLM (or FP8 on H100) is the answer there.

Pick this if…

• You're using Ollama or LM Studio: ★ ★ Q4_K_M, done.
• You want highest quality on one 24GB consumer GPU: EXL2 4.5–5.0bpw via TabbyAPI.
• You're running vLLM in production: AWQ-4bit or FP8 if hardware supports.
• You have an H100 / H200 / RTX 4090+: FP8 is excellent.
• You're squeezing a 405B model into a Mac Studio: Q4_K_S or even Q3_K_M (large MoEs quantize better).