Home Assistant LLM Integrations

The 2026 differentiator. "Hey Jarvis, dim the lights and tell me how the spaghetti's coming along" goes to a local Whisper, then a local Ollama running Llama 3.3, which selects from your exposed HA entities and calls light.turn_on and timer.start, then Piper says it back. All locally. Not science fiction; widely deployed.

For HA itself see Smart Home Hubs; for voice infra (wakeword / STT / TTS) see Smart Home Voice; for the broader LLM landscape see AI / LLM; for self-hosting LLMs at the OS level see Self-Hosted AI / LLM.

What HA actually does with an LLM

HA's "Conversation" subsystem hands the LLM:

1. Your spoken text (post-Whisper).
2. A list of HA entities you've exposed.
3. A list of available HA tool calls (turn_on, set_temperature, set_position, etc.).

The LLM picks an entity + service call OR generates a chat reply. HA executes the call. Piper speaks the response. Function-calling LLMs are dramatically better at this than chat-only models.

Local LLM runners

• ★ Ollama — single-binary local LLM runtime; serves Llama / Mistral / Qwen / Gemma / Phi etc. via HTTP API. HA has a first-class Ollama Conversation integration. The default for local LLM in HA.
  • https://github.com/ollama/ollama
• ★ Ollama HA add-on — community add-on; installs Ollama on the HA host; convenient for "I have a NUC with a GPU."
• llama.cpp / llama-server — direct llama.cpp HTTP server; supports more model formats; HA integration via OpenAI-compatible endpoint.
• LM Studio — desktop GUI runner; good for development; OpenAI-compatible endpoint that HA can hit.
• vLLM — production-grade LLM server (Python); higher throughput than Ollama for many concurrent users; overkill for single-house HA.
• Text Generation WebUI / Open WebUI — UI on top of llama.cpp / Ollama; useful for chat outside HA.

For the deeper picture see Self-Hosted AI / LLM.

Models that work well for HA Conversation

In 2026, the sweet-spot tier:

• ★ Llama 3.3 8B / Mistral Nemo 12B / Qwen 2.5 7B Instruct — function-calling capable, ~5-8 GB VRAM at Q4 quant; runs on a 12-16 GB GPU. Fast enough for sub-2 s responses with HA Voice.
• Llama 3.3 70B / Qwen 2.5 32B / Mistral Large — much smarter; needs a workstation GPU (24 GB+) or Mac M-series with 32+ GB unified RAM. Slower but handles ambiguous requests.
• Phi-3.5 / Llama 3.2 3B — tiny; runs on a Pi 5 with 8 GB RAM at painful speed; only useful for simple turn-on / off intents.

Tool-calling fine-tunes specifically tuned for HA-style function calling are a growing community — search "hass-llm-extension" / "ha-mistral" on Hugging Face.

Cloud LLM options (the Conversation integrations)

If local isn't fast/smart enough:

• ★ OpenAI Conversation — built-in HA integration; GPT-4o / 4.1 / 5 (depending on what's current); ~$1-3/mo at home use; the "smart and reliable" easy button.
• ★ Google Generative AI / Gemini Conversation — built-in; Gemini 1.5/2.5; generous free tier; multimodal (handles camera images).
• Anthropic / Claude Conversation — community integration; Claude is excellent at HA's function-calling.
• Groq / Together / Fireworks — fast cloud inference for open-weight models; community integrations.
• Nabu Casa Cloud — bundled access to a curated cloud LLM; convenient for subscribers.

Privacy / cost trade-off: local Ollama is private and free after hardware; cloud LLMs are smarter and cheaper than their hardware equivalent (a 4090 isn't free).

The full local Voice + LLM stack

The 2026 reference build:

1. HA Voice PE / ESP32-S3-BOX — wakeword detection (microWakeWord on-device).
2. Whisper add-on — faster-whisper-medium-int8 on a CPU is fine for English; large-v3 on GPU for non-English.
3. Ollama add-on — Llama 3.3 8B Instruct, Q4 quant, 6 GB VRAM. HA Conversation pointed at it.
4. Piper add-on — en_US-lessac-medium voice; sub-second TTS.
5. HA Assist Pipeline — chains everything; configurable per device.

Latency budget: wakeword (instant) → audio capture (~1 s) → STT (~0.5-1.5 s) → LLM (~1-3 s) → TTS (~0.3 s). End-to-end: 2.5-5 seconds on a Pi 5 + Coral; <1.5 s on a 4060+ GPU.

Vision LLMs (the camera integration)

• ★ LLaVA / Qwen2-VL / Llama 3.2 11B Vision via Ollama — pass a Frigate snapshot, ask "what's at the door." HA llmvision custom integration wires this up.
  • https://github.com/valentinfrlch/ha-llmvision
• ★ GPT-4o / Gemini Pro Vision — cloud option; faster and smarter; pennies per call at home use.
• Pairs naturally with Frigate for "describe what the camera saw."

Custom intent / template scripts

• HA's intent_script + conversation custom sentences — define your own "when the user says X, run this script" without an LLM. Faster, no inference cost; useful for the 80 % of "lights off, kitchen" requests.
• The 2026 best practice: simple sentences → custom intents (instant); complex sentences → LLM. HA's pipeline can fall back automatically.

Observability / debugging

• Voice Pipeline → "Show debug" in HA — see exactly what STT, the LLM, and TTS each emitted. Crucial when "Hey Jarvis, set the kitchen timer for 10 minutes" misfires.
• Ollama logs — ollama logs; check token counts and inference time per call.
• Long-context drift — multi-turn conversations can confuse smaller LLMs; reset history liberally.

Practical guidance

• ★ Expose the right entities, not all of them. A 200-entity prompt confuses the LLM and bloats inference time. Use HA's per-entity "Expose to Assist" toggle.
• ★ Use shorter, more specific entity names. "Kitchen ceiling light" beats "Lutron RA2 Switch in Kitchen."
• Function-calling models destroy chat-only models for HA tasks. Verify your model supports tools before evaluating quality.
• Cloud LLM for "fall-back when local can't" — HA's pipeline supports this natively.
• Don't expose unsafe entities. Garage door / lock entities should require confirmation; HA has a "require confirmation" toggle.

Pick this if…

• Default local LLM: Ollama + Llama 3.3 8B Instruct.
• Smarter, smartest: GPT-4o or Claude via Conversation integration.
• Vision (describe-the-camera): GPT-4o or local LLaVA via ha-llmvision.
• Privacy-first: local Ollama on a NUC with a 4060.
• Just starting: built-in OpenAI Conversation + Voice PE; total cost <$100 + ~$2/mo.