Vision and Multimodal Models

Vision-language models (VLMs) — models that take images plus text and produce text — are mature enough to self-host in 2026. The capability gap to GPT-4o-class is bigger than the text-only gap, but useful local VLMs exist at every hardware tier.

For text-only see models overview; for hardware see tiers; for inference see Ollama and vLLM / SGLang.

The picks

★ ★ Qwen2.5-VL / Qwen3-VL

• Apache 2.0; sizes from 3B to 72B.
• Excellent OCR and document understanding; strong on charts, tables, screenshots.
• Native video input (frame-sampled) on 7B+.
• ★ ★ Best-balanced local VLM in May 2026 — license, capability, hardware fit.

★ Llama 3.2 Vision and Llama 4 Vision

• 11B and 90B variants from Meta.
• Llama 4 vision variants release alongside the text models with strong general performance.
• License: Llama Community License.
• Good general capability; less strong on dense OCR than Qwen2.5-VL.

★ Pixtral (Mistral)

• Pixtral 12B; native multimodal architecture; Apache 2.0.
• Strong on natural-image reasoning.
• Good fit for Tier 1+ hardware.

★ MiniCPM-V

• 2.6 / 2.5 series; small (8B) but punches above weight.
• Excellent for "I want a vision model on a Tier-0/1 box."
• Apache 2.0.

★ InternVL 2.5 / 3

• OpenGVLab; 1B–78B; strong general VLM.
• MIT license.

LLaVA (legacy)

• The original open VLM family. Still around; mostly superseded by the above.

Florence-2 (Microsoft)

• Microsoft's small (~770M) multitask vision model. Not a chat model — outputs structured detection / captioning. Great for pipeline use.

OCR-specialised

• ★ GOT-OCR2 — small, dedicated OCR; outperforms general VLMs on dense text.
• PaddleOCR — open-source OCR pipeline; not LLM-based but production-grade.
• For more, see ocr-vision.

What works in 2026

• Ask about an image — describe, count things, identify, extract text. All local VLMs handle this.
• Read screenshots — UI inspection, error-message extraction; Qwen2.5-VL is best.
• Chart / table / document understanding — Qwen2.5-VL leads.
• Code from a screenshot of a UI — works but quality drops vs. closed-frontier models.
• Video understanding — frame-sampled summarisation works on Qwen2.5-VL 7B+; latency-heavy.

What's still hard

• Dense OCR of low-resolution scans. Specialist OCR (GOT-OCR2, PaddleOCR) beats general VLMs.
• Long video understanding. Most VLMs sample 4–32 frames; for actual 30-min video, capability degrades.
• Spatial reasoning ("is the cat to the left of the dog?") — improving but inconsistent.
• Image generation. VLMs are vision input only — for output you want ComfyUI.

Inference engines for VLMs

• Ollama supports llava, qwen2.5-vl, llama3.2-vision, minicpm-v, etc. — pull and run.
• vLLM has growing VLM support — check the model-support matrix.
• llama.cpp GGUF for VLMs requires the right multimodal projector files alongside the GGUF.
• Transformers — HF library, useful for one-off scripting.

Pick this if…

• One default VLM pick for a 24GB GPU: ★ ★ Qwen2.5-VL 32B (or 72B if you have the room).
• Tier 1 / smaller hardware: Qwen2.5-VL 7B or MiniCPM-V.
• OCR-first: GOT-OCR2 + a general VLM as fallback.
• Long-form video understanding: rent an H100 on vast.ai and run a frontier VLM there.
• Frontier capability: the gap to GPT-4o / Claude Opus / Gemini Vision is still real for hard reasoning over images. For the hardest tasks, paid APIs win — for routine vision, local is fine.