How we index images for RAG

Author: Matteo Bortoletto | Published: 2026-06-01 | Generated: 2026-06-02 | Domain: kapa.ai Tags: ‘#rag’ ‘#multimodal-ai’ ‘#image-indexing’ ‘#technical-documentation’ ‘#retrieval’

---

TLDR

Kapa’s RAG pipeline processes screenshots, diagrams, and tables once during ingestion: a vision-language model produces a textual caption or transcription, which is stored as a separate retrievable chunk alongside documentation text. This avoids the cost, payload limits, and weak fine-grained retrieval of query-time multimodal pipelines while preserving image-derived facts and actionable visual context. Across three customer projects, caption-enabled retrieval significantly improved LLM-judged answer quality at just 1–6% additional per-query cost.

Key Takeaways

• Index images once, retrieve text thereafter: Vision models caption or transcribe each image at ingestion; at query time the LLM receives relevant caption chunks and cites the original image URL without processing raw pixels.
• Query-time multimodal RAG is costly and constrained: Raw images increased per-query cost by 27% with GPT 5.1 and 51% with Claude 4.6 Sonnet; Claude tokenizes an image at roughly 975 tokens, versus GPT’s 716. Typical retrieval references 20–30 images, with a long tail beyond 130, nearing Claude’s 30 MB payload limit at about 25 images.
• Captions preserve both illustrative and load-bearing information: Screenshots make instructions more actionable, while transcribed tables, matrices, and schematics expose facts that may exist nowhere in surrounding prose.
• Filtering needs pragmatic tradeoffs: Heuristics remove unsupported, tiny, or unusually shaped images; a zero-shot multimodal-embedding classifier achieved 96.8% accuracy (F1 0.974) on clear cases and removed about 13% of post-heuristic images, but ambiguous images reached only 59.8% accuracy without textual context.
• Separate caption chunks outperform inline captions: On an image-heavy project, inline captions increased GPT query cost by 19%, versus 6% for separate chunks. Captions reached the reranker’s top 15 on 51% of queries while overall ranking remained stable (Spearman ρ = 0.905); image placement was correct 94–99% of the time.

Images & Media

• Article header illustration — Introductory visual for Kapa’s image-indexing approach.
• Answer comparison screenshot — Contrasts a text-only answer with one that includes an actionable supporting screenshot.
• Image-context quality illustration — Demonstrates how visual context makes support answers easier to follow.
• Load-bearing datasheet figure — Example of a technical figure whose table or diagram content can be the answer itself.
• Image-filtering illustration — Visual context for filtering noise such as logos, banners, and decorative images before captioning.

Referenced Links

• Hacker News discussion — Community discussion of the article.
• Microsoft: Multimodal RAG with Vision — Microsoft research/blog post supporting ingestion-time image description and separate storage.