Local RAG Stacks

Retrieval-Augmented Generation (RAG) — "ask my documents" — is the second most common self-hosted AI workload after chat. The local FOSS stack matured significantly in 2024–25 with GraphRAG (Microsoft, late 2024), LightRAG (2024), and improvements to existing tools. For production-grade RAG at home you want: a good embeddings model, a vector store, a reranker, and a UI/orchestration layer.

Cross-links: embeddings for the model picks; Open WebUI for the chat UI with built-in RAG; AnythingLLM for RAG-first; Ollama for the LLM layer; hardware tiers for what runs.

The components of a local RAG stack

1. Documents in. PDF / DOCX / HTML / Markdown / scanned-OCR.
2. Chunker. Split into 256–1024 token chunks (semantic / recursive / hierarchical).
3. Embedder. Turn each chunk into a vector — see embeddings.
4. Vector store. Index the vectors for nearest-neighbour search.
5. Retriever. At query time: embed query, find top-K chunks; optionally rerank.
6. Generator. Stuff chunks into the LLM context with the question.

Tools that bundle the whole pipeline

★ ★ Open WebUI

• Built-in RAG over knowledge bases.
• File upload + URL scraping + YouTube ingestion.
• Hybrid keyword + vector search.
• Reranker support.
• Default if you also want a chat UI.

★ AnythingLLM

• RAG-first; workspaces are knowledge bases.
• Many embedders / vector DBs (pgvector, Qdrant, Chroma, Weaviate, Pinecone, Milvus, LanceDB).
• Document upload UI; metadata filtering.
• Default if RAG is the headline feature.

★ Khoj

• Personal AI for your knowledge — Obsidian / Notion / Markdown / PDF / org-mode.
• Self-host or use their hosted; FOSS.
• Strong PKM-meets-RAG story.

★ LightRAG

• Microsoft Research's lighter alternative to GraphRAG.
• Builds entity graphs from documents for better multi-hop retrieval.
• Apache 2.0.

★ GraphRAG (Microsoft)

• Knowledge-graph-based RAG; handles "summarize this whole corpus" queries that vector RAG fails at.
• More compute-intensive; requires building a graph upfront.
• MIT.

Verba (Weaviate)

• Open-source RAG app; nice UI; Weaviate-native.

LlamaIndex / LangChain

• Frameworks for assembling your own RAG pipeline. More flexibility, more work.

Picking embeddings (the most-impactful choice)

See embeddings deep dive. Quick picks:

• ★ ★ bge-m3 — multilingual, long context (8K), strong default.
• ★ nomic-embed-text-v1.5 — Apache 2.0, Matryoshka.
• ★ mxbai-embed-large — strong English.

Picking a vector DB

• ★ ★ pgvector — Postgres extension. If you already run Postgres, just add this.
• ★ ★ Qdrant — Rust; great filtering; standalone.
• ★ Chroma — Python-friendly; good for prototyping.
• ★ LanceDB — file-based; embedded in apps.
• Milvus — high-scale multi-tenant.
• Weaviate — vector + ML modules; more opinionated.

Reranking

A reranker after vector search is a major quality lift. ★ ★ bge-reranker-v2-m3 is the canonical pick; runs locally; small compute cost.

Hybrid search (BM25 + vector)

Pure vector search misses keyword matches; pure BM25 misses semantic. Hybrid is meaningfully better than either. Open WebUI does this; AnythingLLM does this; if rolling your own, use Postgres pgvector + pg_trgm or Qdrant's hybrid mode.

Document parsing

The unsexy but most-impactful step:

• ★ ★ Docling (IBM) — strong PDF/DOCX/HTML parser; layout-aware.
• ★ Unstructured — broad format support; MIT/Apache mix per component.
• ★ MarkItDown (Microsoft) — converts office docs to markdown.
• PyMuPDF / pdfplumber / pypdf — direct PDF libraries.
• For OCR see ocr-vision; for the broader doc-parsing landscape see document-parsing-rag.

Chunking strategies

• Fixed-size (e.g., 512 tokens with 50 overlap) — simple, often fine.
• Recursive character — respects markdown / code structure.
• Semantic — splits at sentence-similarity drops.
• Hierarchical / parent-child — small chunks for retrieval, parent chunks for context.
• Late chunking (Jina, 2024) — embed full doc, chunk after.

For most use cases, recursive character at 512 tokens with 50 overlap is a fine default. Optimize from there.

Honest pitfalls

• Cheap embeddings give cheap RAG. Use a real embedder (bge-m3, mxbai-embed-large), not the tiniest model.
• Skipping reranking. Add a reranker; the lift is large.
• Ignoring metadata. Source filenames, dates, sections — filter on these.
• Too-small chunks. 128-token chunks lose context; 1024+ is often better than people assume.
• Too-much context. Stuffing 50 chunks into a context degrades answer quality; 5–10 well-ranked chunks beat 50 mediocre.

Picking based on use case

• "Ask my Obsidian notes": Khoj or Open WebUI with the vault as a knowledge base.
• "Ask scanned documents from Paperless-ngx": Open WebUI / AnythingLLM with a Paperless-ngx connector or MCP server.
• "Customer support over docs": AnythingLLM workspaces + reranker.
• "Summarise a whole book": GraphRAG or LightRAG, not vector RAG alone.
• "Code Q&A over a repo": Continue.dev's @codebase or Aider's repo-map.
• "Ask a NotebookLM-style podcast": Open NotebookLM (see Big-AGI / Msty).

Pick this if…

• One bundled RAG tool, you also want chat UI: ★ ★ Open WebUI.
• One bundled RAG tool, RAG-first: ★ ★ AnythingLLM.
• PKM + AI (Obsidian / org-mode): Khoj.
• Multi-hop / graph queries: LightRAG or GraphRAG.
• Roll-your-own with control: LlamaIndex or LangChain + Qdrant + bge-m3 + bge-reranker-v2-m3.
• Chat over codebase: Continue.dev's @codebase or Aider repo map.