Hardware Tier Guide

The headline page of the self-hosted AI section. Real prices, real recommendations, real models that actually run at each tier in May 2026. The honest version: for chat-quality use you can get going for $0 on what you already own; for serious-quality you want a used $700–900 RTX 3090; for "feels like ChatGPT-3.5 fast" you want a used 3090 or new RTX 5090; for "feels like GPT-4-of-2023 fast at home" you're spending $3,500–8,000.

Cross-links throughout: Ollama is the install path for most tiers; models for what runs at each size; quantization for understanding Q4 vs Q8 vs FP16; Mac for the Apple Silicon track; cloud rentals for "rent before you buy."

Reading the tiers

• VRAM is the bottleneck for GPU inference. A model needs roughly (parameters × bytes-per-param) + a few GB of overhead of VRAM. A 7B model at Q4 is ~5GB; at Q8, ~9GB. A 70B model at Q4 is ~42GB; at Q8, ~75GB.
• System RAM is the bottleneck for CPU inference, and matters for Apple Silicon (unified memory).
• Tokens/sec numbers below are rough chat-style generation rates. Prompt processing (the "thinking" pause before reply starts) is separate and Apple Silicon is notably slower at it.
• Q4_K_M is the default sweet-spot quant — see ai-selfhost-quantization-gguf-awq.

Tier 0 — $0–200: existing hardware, get started

Goal: prove the workflow, run small models on whatever you have.

• CPU + 16GB RAM, no GPU — works. Slow but works. Ollama on CPU runs Llama 3.2 3B or Phi-4-mini at 4–10 tokens/sec. Useful for: testing the workflow, autocomplete on small files, getting comfortable with Ollama / Open WebUI before buying hardware.
• Existing iGPU (Intel Arc, AMD RDNA3 iGPU) — partial GPU acceleration via Vulkan in llama.cpp; modest speedup over pure CPU.
• Old gaming GPU you already own (GTX 1080, RTX 2060) — use it. Even 6–8GB VRAM lets a 7B-Q4 model run at 30+ tokens/sec.
• What runs well: 1B–3B models (Llama 3.2 1B/3B, Phi-4-mini, Gemma 3 4B, Qwen 3 1.5B/3B). 7B-Q4 models on CPU are slow but usable for short chats.
• What doesn't: 13B+ painfully slow on CPU; 30B+ measured in seconds-per-token, not tokens-per-second.

Honest verdict: start here for free; you'll quickly know whether you want to invest. If you already have any modern GPU, that's all you need to begin.

Tier 1 — $300–700: first real upgrade

Goal: comfortably run 7B–8B models, taste 13B, decide what bothers you.

The three paths at this tier:

• Used RTX 3060 12GB (~$200–250 used as of May 2026) — the budget VRAM king for years; 12GB fits 13B-Q4 with room. ~30 tok/s on 7B-Q4, ~15 tok/s on 13B-Q4.
• Used RTX 3060 Ti 8GB / RTX 3070 8GB (~$180–250) — faster compute than the 3060 12GB but only 8GB VRAM, capping you at 7B-Q4 / 8B-Q4.
• CPU + 32–64GB DDR5 (~$300 RAM upgrade for an existing PC) — runs 30B-Q4 on CPU at 2–4 tok/s; tolerable for batch tasks, painful for chat. Useful if you mostly want a "nightly summarizer" that doesn't need real-time speed.
• Mac Mini M4 base, 16GB unified (~$600 new, May 2026) — 7B-Q4 at ~25 tok/s. Tier-1-equivalent in a desk-toy form factor.
• Used Mac Mini M2/M2 Pro, 16–32GB (~$500–800 used) — same story, more RAM headroom.

What runs well: Llama 3.2 8B, Llama 4-mini-class, Qwen 3 7B / 8B, Mistral 7B, Gemma 3 9B, Phi-4 14B at Q4. Qwen2.5-Coder 7B for autocomplete is genuinely useful here.

What doesn't: 30B at any reasonable speed; 70B at all on a single 12GB GPU.

Honest verdict: Tier 1 is the sweet spot for "I want to try this seriously without committing." The used RTX 3060 12GB is the canonical first GPU for local LLMs in 2026, and has been since 2022.

Tier 2 — $800–1500: mid-range, the practical home setup

Goal: run 30B-Q4 comfortably; touch 70B-Q4 slowly.

• Used RTX 3090 24GB (~$700–900 used as of May 2026) — ★ ★ the canonical local-LLM GPU. 24GB VRAM fits 30B-Q4 with room or 70B-Q4 if quantized hard. ~40 tok/s on 8B, ~25 tok/s on 30B-Q4, ~8–12 tok/s on 70B-Q4. Power-hungry (350W TDP) and bulky but unbeatable price/perf for inference.
• RTX 4060 Ti 16GB (~$450 new) — modern, efficient (165W), 16GB fits 13B comfortably and 30B-Q4 just barely. Slower than 3090 on 30B+. Good if power / heat / noise matter.
• RTX 4070 Super 12GB (~$550) — fast for 7B–13B; cramped for 30B.
• Apple Silicon Mac Mini M4 Pro 24GB / 48GB ($1,400–1,800 new) — comparable to 3090 for 30B model fit, slower prompt processing, much quieter and lower-power.
• Build a tower around a used 3090: $400 mobo+CPU+RAM+PSU+case + $800 used 3090 = ~$1,200 build. Best Tier-2 outcome.

What runs well: Qwen 3 32B, DeepSeek-V2-Lite, Qwen2.5-Coder 32B (the local coding champion), Llama 3.3 70B-Q4 slowly, Mistral Small 3, Gemma 3 27B. Llama 4-Scout-class at Q4.

What doesn't: 70B at full FP16 quality; 100B+ models; multiple agents in parallel.

Honest verdict: This tier is where local AI stops feeling like a science project. Used 3090 + 64GB system RAM + a fast NVMe is the no-brainer build. ★ ★ recommendation if you're actually committing.

Tier 3 — $1500–3500: serious local AI

Goal: run 70B-Q4 fast; run 30B at full Q8 / FP16; do real work.

• RTX 4090 24GB ($1,600–2,000 new May 2026) — the consumer flagship of the 40-series; same VRAM as 3090 but ~40% faster compute and FlashAttention-3 support. ~70 tok/s on 8B, ~40 tok/s on 30B-Q4, ~15–20 tok/s on 70B-Q4.
• RTX 5090 32GB ($2,000–2,800 new, released early 2026) — ★ the new flagship. 32GB VRAM lets 70B-Q4 fit comfortably with room for context, and runs ~30% faster than the 4090. The first single-card consumer "I can run 70B locally" answer.
• 2× used RTX 3090 (~$1,400–1,800 for the pair + $300 motherboard with proper PCIe lanes + PSU upgrade) — 48GB total VRAM via tensor parallelism in vLLM / SGLang. Runs 70B-Q8 (full quality), or 100B+ at Q4. The classic "DIY local AI" build.
• AMD Radeon RX 7900 XTX 24GB ($800–950) — ROCm has finally reached "mostly works" for llama.cpp / Ollama by 2026, but still trails NVIDIA on bleeding-edge frameworks. Good budget VRAM if you tolerate slightly more setup pain.
• Mac Studio M3 Max 64GB ($3,000 new) or Mac Studio M3 Ultra 96GB ($4,800) — runs 70B-Q4 at 10–15 tok/s, 70B-Q8 at 5–8 tok/s, and is silent. Slower prompt processing than 3090 but much more RAM headroom.

What runs well: Llama 4 70B-class at Q4 (≈ GPT-4-of-2023 quality at home), Qwen 3 72B, DeepSeek V3 medium quants, Mistral Large 2 quantized, Command-R+ at Q4. Multiple agents in parallel via vLLM batching.

What doesn't: 405B models without aggressive quantization; serious training / fine-tuning on a single card.

Honest verdict: Tier 3 is where local meaningfully competes with API for most workloads. RTX 5090 32GB is the new ★ ★ "single card that does everything" pick if you're buying new in May 2026; dual 3090 is still the price/perf champion if you're patient with a build.

Tier 4 — $3500–8000: workstation, enthusiast pro

Goal: 70B at full Q8/FP16, fine-tune 8B–13B at home, run multiple large models simultaneously.

• Threadripper / EPYC + 128–256GB DDR5 ECC + 2× RTX 4090 / 5090 — $5,000–7,500 build. Real PCIe Gen 5 lanes, real cooling, real noise. The home-lab end of the spectrum.
• NVIDIA RTX 6000 Ada 48GB (~$5,000–6,500) — single-card 48GB; pro driver; quieter than dual 4090s; the "professional workstation" answer.
• NVIDIA RTX A6000 48GB (used, ~$3,500–4,200) — previous-gen pro card; still excellent for inference; the price/perf winner in this tier if you find one.
• Mac Studio M3 Ultra 192GB / 256GB (~$6,000–9,000) — runs 405B models at Q4. Slower than dual 4090s on prompt processing but the RAM headroom is unmatched at this price.

What runs well: Llama 4 405B-class at Q4 (with Mac Studio Ultra or 96GB+ VRAM), 70B-Q8 fast, fine-tuning 7B–13B with QLoRA via Unsloth or Axolotl, multiple production-quality agents in parallel.

What doesn't: Full-precision fine-tuning of 70B; training from scratch.

Honest verdict: This is the "I do AI as a serious hobby or a small-business workload" tier. Most home users skip past this to a managed API. If you're considering it, rent equivalent hardware on vast.ai / RunPod for a month first to validate.

Tier 5 — $8000–25000+: production / fine-tuning

Goal: serious training, fine-tuning at scale, on-prem business deployment.

• NVIDIA H100 80GB (~$25,000–30,000 new, ~$15,000–20,000 used as of May 2026) — the production AI standard. PCIe or SXM5. Most home users do not need this and should rent on demand.
• NVIDIA H200 141GB (~$30,000+) — H100 successor, more memory; mostly relevant for production multi-tenant serving.
• AMD MI300X 192GB — competitive enterprise option; ROCm software still trails CUDA in tooling maturity.
• NVIDIA L40S 48GB (~$8,000–10,000) — datacenter card, great for inference at scale; runs cool in a rack, no display output.
• Used DGX-class servers — surplus enterprise gear surfaces on the second-hand market; expensive to power, loud, and you'll need a 240V circuit and a properly cooled room.

Honest verdict: ★ Almost no home user belongs here. Rent on RunPod / vast.ai / Lambda at $1–4/hr instead. Buy only if you have a continuous production workload or are doing meaningful training research.

Apple Silicon track (cross-tier)

The unified-memory advantage is genuinely different from NVIDIA. RAM size = model size you can load.

• Mac Mini M4 16GB (~$600) — Tier-1-equivalent, runs 7B–8B comfortably.
• Mac Mini M4 Pro 24GB / 48GB ($1,400–1,800) — Tier-2-equivalent, 30B fits.
• MacBook Pro M3 Pro 36GB / Max 48GB ($2,500–3,500) — laptop-portable Tier-2/3.
• Mac Studio M3 Max 64GB (~$3,000) — runs 70B-Q4 well, silent.
• Mac Studio M3 Ultra 96GB / 192GB / 256GB ($4,800–9,000) — runs 70B-Q8, 200B-Q4, 405B-Q4 respectively. Unmatched RAM-per-dollar at the top end.

Honest tradeoffs:

• Pro: silent, low-power, unified memory means you load whatever fits in RAM, MLX framework is now mature, no driver hell.
• Con: prompt processing is 2–4× slower than equivalent NVIDIA on long contexts. Generation is competitive. The 30-second wait before a 30K-token context starts replying is the worst part.
• Pick a Mac if: you also use the machine for non-AI work, you want silence, you live in a small apartment, you value the platform.
• Pick NVIDIA if: prompt-processing speed matters, you're doing long-context RAG or agentic loops, you want to fine-tune.

See ai-selfhost-mac-apple-silicon for the Mac-specific deep dive.

Cloud rental tier (rent before you buy)

• vast.ai — cheapest hourly GPU rental marketplace. RTX 3090 ~$0.20–0.30/hr; RTX 4090 ~$0.40–0.60/hr; H100 ~$1.80–2.50/hr.
• RunPod — slicker UI, persistent volumes, "community cloud" cheap tier and "secure cloud" slightly pricier.
• Lambda Labs — premium, US-based, great for production training.
• TensorDock, Modal, Hyperstack — newer entrants.

Use cloud rentals to:

• Validate the build — rent a 3090 on vast.ai for a week, run your real workload, see if it's enough, before dropping $900.
• Spike workloads — fine-tuning, big batch jobs, occasional 70B+ inference.
• Avoid hardware entirely — at light usage, spending $0.20/hr × 4hr/day × 30 days = $24/month beats both buying hardware and most paid APIs.

See ai-selfhost-cloud-rentals-runpod-vastai for the deep dive.

Things people get wrong at every tier

• Buying for VRAM only. A 16GB RTX 4060 Ti is slower than a 12GB RTX 3060 12GB for some workloads — VRAM is necessary but not sufficient. Compute and memory bandwidth matter too.
• Underspeccing power and cooling. A 3090 needs a real 850W PSU and real airflow. Two 3090s need 1200W and serious cooling. Don't skimp.
• Buying brand-new when used is fine. RTX 3090 used at $800 outperforms RTX 4070 new at $600 for inference. The 30-series is an excellent used buy in 2026.
• Forgetting about disk. Open weights are big. A serious local-AI box wants 2TB NVMe minimum.
• Forgetting about noise. A 4090 at full tilt is loud. A pro card (RTX 6000 Ada / A6000) is quieter. A Mac Studio is silent.
• Not budgeting for power. A 3090 at full inference 24/7 draws $20–50/month in electricity at US rates. Heat too — a small room with two 3090s is genuinely uncomfortable in summer.

Pick this if…

• Free / "what I have": Tier 0. Install Ollama, run Llama 3.2 3B.
• Cheapest credible tier: Tier 1 with a used RTX 3060 12GB ($220), or a Mac Mini M4 base.
• Best price/perf in 2026: ★ ★ Tier 2 with a used RTX 3090 ($800–900). The canonical local-LLM build.
• One single card that does everything new: Tier 3 with an RTX 5090 32GB ($2,500).
• 70B at full quality on a single card: Tier 4 with RTX 6000 Ada 48GB or used A6000.
• 400B+ models at home: Mac Studio M3 Ultra 256GB.
• Serious training: rent H100s on vast.ai or Lambda.
• Silent + low power, doesn't mind slow prompt processing: any Apple Silicon path.
• Not sure: rent a 3090 on vast.ai for $30, prove the workflow, then buy.