Abstract: Retrieval over visually rich documents is essential for tasks such as legal discovery, scientific search, and enterprise knowledge management. Existing approaches fall into two paradigms: single-vector retrieval, which is efficient but coarse, and multi-vector retrieval, which is accurate but computationally expensive. To address this trade-off, we propose HEAVEN, a two-stage hybrid-vector framework. In the first stage, HEAVEN efficiently retrieves candidate pages using a single-vector method over Visually-Summarized Pages (VS-Pages), which assemble representative visual layouts from multiple pages. In the second stage, it reranks candidates with a multi-vector method while filtering query tokens by linguistic importance to reduce redundant computations. To evaluate retrieval systems under realistic conditions, we also introduce ViMDOC, the first benchmark for visually rich, multi-document, and long-document retrieval. Across four benchmarks, HEAVEN attains 99.87% of the Recall@1 performance of multi-vector models on average while reducing per-query computation by 99.82%, achieving efficiency and accuracy.
The Efficiency-Accuracy Dilemma in Visual Document Retrieval
In Retrieval-Augmented Generation (RAG), finding the right information in "visually rich" documents—such as scientific papers, legal filings, or financial reports—is notoriously difficult. Standard text-based retrieval misses the context of charts and layouts, while newer Visual Document Retrieval (VDR) methods face a dilemma between two paradigms:
- Single-Vector Retrieval: Encodes a whole page into one vector. It is fast but often misses fine-grained details needed for precise matching.
- Multi-Vector Retrieval: Encodes pages into hundreds of token-level or patch-level embeddings (e.g., ColPali). It is highly accurate due to "late interaction" between query terms and specific image patches, but it is computationally expensive and slow for large-scale use.
Analysis shows that while single-vector models struggle to pick the #1 best page, they are effective at ensuring the correct page is within the top 100 or 200 candidates. The challenge is bridging this gap without the massive compute overhead of full multi-vector scanning.
The HEAVEN Framework
HEAVEN (Hybrid-vector retrieval for Efficient and Accurate Visual multi-documENt) is a two-stage pipeline designed to solve this trade-off.
Stage 1: Efficient Filtering with VS-Pages
To avoid scanning every single page in a massive corpus, HEAVEN creates Visually-Summarized Pages (VS-Pages). It uses Document Layout Analysis (DLA) to extract titles and headers from multiple consecutive pages and assembles them into a single "summary" image. A fast single-vector model (DSE) searches these summaries first to identify candidate documents, then narrows the search down to specific pages.
Stage 2: Precise Reranking with Token Filtering
Once a small set of candidate pages is identified, a multi-vector model (ColQwen2.5) takes over. To further save compute, HEAVEN performs Query Token Filtering. It uses Part-of-Speech (POS) tagging to identify "key tokens" (nouns and entities). It performs the initial rerank using only these high-signal tokens—reducing cross-attention calculations by approximately 70%—before a final pass with the full query on the top-most candidates.
ViMDOC: A Realistic Benchmark
The researchers introduced ViMDOC (Visually-rich Long Multi-Document Retrieval) to evaluate systems under realistic conditions. Unlike existing datasets that focus on single, short documents, ViMDOC features:
- Long documents: Averaging over 50 pages.
- Multi-document queries: Questions requiring information scattered across different files.
- Large scale: Over 76,000 pages in the corpus.
Key Results and Insights
HEAVEN achieves a state-of-the-art balance between performance and computational cost. By leveraging the "acceptable coarseness" of single-vectors for initial filtering and linguistic importance for visual search, it outperforms existing pruning methods.
- Massive Efficiency Gains: HEAVEN reduced per-query computation (FLOPs) by 99.82% compared to the full ColQwen2.5 multi-vector model.
- Near-Lossless Accuracy: It retained 99.87% of the multi-vector model's Recall@1 performance on average across four benchmarks.
- Semantic Compression: VS-Pages allow the system to "look" at 15 pages of content in the time it usually takes to look at one, without losing navigational cues.
While HEAVEN's efficiency relies on the quality of layout analysis, it demonstrates that hybrid-vector approaches are the path forward for scalable, high-accuracy retrieval in visually rich domains.
Summary updated