TL;DR
HEAVEN is a new two-stage retrieval framework that solves the "efficiency vs. accuracy" trade-off in searching visually rich documents (like PDFs with charts and tables). By combining coarse single-vector filtering with a refined multi-vector reranking process, it achieves 99.87% of the accuracy of state-of-the-art models while slashing computational costs (FLOPs) by over 99.8%.
What Problem Does This Paper Address?
In the world of Retrieval-Augmented Generation (RAG), finding the right information in "visually rich" documents—scientific papers, legal filings, or financial reports—is notoriously difficult. Standard text-based retrieval misses the context of charts and layouts, while newer Visual Document Retrieval (VDR) methods face a dilemma:
- Single-Vector Retrieval: Encodes a whole page into one vector. It’s incredibly fast but "blunt," often missing fine-grained details needed for precise matching.
- Multi-Vector Retrieval: Encodes pages into hundreds of token-level or patch-level embeddings (e.g., ColPali). It’s highly accurate because it allows for "late interaction" between query terms and specific image patches, but it is computationally expensive and slow for large-scale use.
The authors identify that while multi-vector models are superior at finding the exact right page (Recall@1), single-vector models are actually quite decent at finding a neighborhood of relevant pages (Recall@200). The challenge is bridging this gap without the massive compute overhead.
Methods
The authors propose HEAVEN (Hybrid-vector retrieval for Efficient and Accurate Visual multi-documENt), a two-stage pipeline:
- Stage 1: Efficient Filtering with VS-Pages. To avoid scanning every single page in a massive corpus, HEAVEN creates Visually-Summarized Pages (VS-Pages). It uses layout analysis to extract titles and headers from multiple consecutive pages and assembles them into a single "summary" image. A fast single-vector model (DSE) searches these summaries first to identify candidate documents, then narrows it down to specific pages.
- Stage 2: Precise Reranking with Token Filtering. Once a small set of candidate pages is identified, a heavy-duty multi-vector model (ColQwen2.5) takes over. To save even more compute, HEAVEN performs Query Token Filtering. It uses Part-of-Speech (POS) tagging to identify "key tokens" (nouns and entities). It performs the initial rerank using only these high-signal tokens—reducing cross-attention calculations by ~70%—before a final pass with the full query on the top-most candidates.
Datasets
The researchers evaluated HEAVEN on several standard VDR benchmarks: OpenDocVQA, ViDoSeek, and M3DocVQA.
Crucially, they introduced a new benchmark called VIMDOC (Visually-rich Long Multi-Document Retrieval). Unlike existing datasets that often focus on single, short documents, VIMDOC mimics real-world complexity by featuring:
- Long documents: Averaging over 50 pages.
- Multi-document queries: Questions that require retrieving information scattered across different files.
- Large scale: Over 76,000 pages in the corpus.
Key Insights and Ideas
- The "Acceptable Coarseness" of Single-Vectors: The authors proved that while single-vector models struggle to pick the #1 best page, they are very effective at ensuring the correct page is within the top 100 or 200. This makes them perfect "gatekeepers" for a more expensive second stage.
- Linguistic Importance in Visual Search: Even in a purely visual retrieval task (where pages are treated as images), the text of the query still follows linguistic rules. Filtering for nouns and entities removes "noise" tokens (like "what," "is," "the") that otherwise waste millions of FLOPs during the patch-matching process.
- Visual Compression via Layout: VS-Pages act as a form of "semantic compression." By focusing on titles and headers, the system can "look" at 15 pages of content in the time it usually takes to look at one, without losing the navigational cues needed to find the right section.
Key Results
- Massive Efficiency Gains: HEAVEN reduced per-query computation (FLOPs) by 99.82% compared to the full ColQwen2.5 multi-vector model.
- Near-Lossless Accuracy: Despite the speedup, it retained 99.87% of the multi-vector model's Recall@1 performance on average across four benchmarks.
- Superiority over Pruning: Compared to other efficiency methods like "patch pruning" (randomly dropping image patches), HEAVEN’s query-filtering approach maintained much higher accuracy at the same computational budget.
Limitations and Future Work
- Layout Dependency: The efficiency of Stage 1 relies on the quality of the Document Layout Analysis (DLA). If a document has a highly irregular or noisy layout, the "VS-Page" construction might miss critical headers.
- Model Sensitivity: The performance is tied to the underlying Vision-Language Models. As these models scale, the authors note that the "gap" between single and multi-vector performance might shift.
- End-to-End RAG: The current paper focuses strictly on the retrieval phase. Integrating this into a full "Generation" pipeline to see how it impacts final LLM answers is a suggested next step.
Summary updated