Hybrid-Vector Retrieval for Visually Rich Documents: Combining Single-Vector Efficiency and Multi-Vector Accuracy

Juyeon Kim, Geon Lee, Dongwon Choi, Taeuk Kim, Kijung Shin

Abstract: Retrieval over visually rich documents is essential for tasks such as legal discovery, scientific search, and enterprise knowledge management. Existing approaches fall into two paradigms: single-vector retrieval, which is efficient but coarse, and multi-vector retrieval, which is accurate but computationally expensive. To address this trade-off, we propose HEAVEN, a two-stage hybrid-vector framework. In the first stage, HEAVEN efficiently retrieves candidate pages using a single-vector method over Visually-Summarized Pages (VS-Pages), which assemble representative visual layouts from multiple pages. In the second stage, it reranks candidates with a multi-vector method while filtering query tokens by linguistic importance to reduce redundant computations. To evaluate retrieval systems under realistic conditions, we also introduce ViMDOC, the first benchmark for visually rich, multi-document, and long-document retrieval. Across four benchmarks, HEAVEN attains 99.87% of the Recall@1 performance of multi-vector models on average while reducing per-query computation by 99.82%, achieving efficiency and accuracy.

The Efficiency-Accuracy Dilemma in Visual Document Retrieval

In Retrieval-Augmented Generation (RAG), finding the right information in "visually rich" documents—such as scientific papers, legal filings, or financial reports—is notoriously difficult. Standard text-based retrieval misses the context of charts and layouts, while newer Visual Document Retrieval (VDR) methods face a dilemma between two paradigms:

  1. Single-Vector Retrieval: Encodes a whole page into one vector. It is fast but often misses fine-grained details needed for precise matching.
  2. Multi-Vector Retrieval: Encodes pages into hundreds of token-level or patch-level embeddings (e.g., ColPali). It is highly accurate due to "late interaction" between query terms and specific image patches, but it is computationally expensive and slow for large-scale use.

Analysis shows that while single-vector models struggle to pick the #1 best page, they are effective at ensuring the correct page is within the top 100 or 200 candidates. The challenge is bridging this gap without the massive compute overhead of full multi-vector scanning.

The HEAVEN Framework

HEAVEN (Hybrid-vector retrieval for Efficient and Accurate Visual multi-documENt) is a two-stage pipeline designed to solve this trade-off.

Stage 1: Efficient Filtering with VS-Pages

To avoid scanning every single page in a massive corpus, HEAVEN creates Visually-Summarized Pages (VS-Pages). It uses Document Layout Analysis (DLA) to extract titles and headers from multiple consecutive pages and assembles them into a single "summary" image. A fast single-vector model (DSE) searches these summaries first to identify candidate documents, then narrows the search down to specific pages.

Figure 5: The construction process of Visually-Summarized Pages (VS-Pages) using Document Layout Analysis (DLA).

Stage 2: Precise Reranking with Token Filtering

Once a small set of candidate pages is identified, a multi-vector model (ColQwen2.5) takes over. To further save compute, HEAVEN performs Query Token Filtering. It uses Part-of-Speech (POS) tagging to identify "key tokens" (nouns and entities). It performs the initial rerank using only these high-signal tokens—reducing cross-attention calculations by approximately 70%—before a final pass with the full query on the top-most candidates.

ViMDOC: A Realistic Benchmark

The researchers introduced ViMDOC (Visually-rich Long Multi-Document Retrieval) to evaluate systems under realistic conditions. Unlike existing datasets that focus on single, short documents, ViMDOC features:

  • Long documents: Averaging over 50 pages.
  • Multi-document queries: Questions requiring information scattered across different files.
  • Large scale: Over 76,000 pages in the corpus.

Key Results and Insights

HEAVEN achieves a state-of-the-art balance between performance and computational cost. By leveraging the "acceptable coarseness" of single-vectors for initial filtering and linguistic importance for visual search, it outperforms existing pruning methods.

Figure 1: Performance (Recall@1) vs. Computational Cost (FLOPs). HEAVEN achieves near-lossless accuracy compared to multi-vector models while reducing FLOPs by over 99%.
  • Massive Efficiency Gains: HEAVEN reduced per-query computation (FLOPs) by 99.82% compared to the full ColQwen2.5 multi-vector model.
  • Near-Lossless Accuracy: It retained 99.87% of the multi-vector model's Recall@1 performance on average across four benchmarks.
  • Semantic Compression: VS-Pages allow the system to "look" at 15 pages of content in the time it usually takes to look at one, without losing navigational cues.

While HEAVEN's efficiency relies on the quality of layout analysis, it demonstrates that hybrid-vector approaches are the path forward for scalable, high-accuracy retrieval in visually rich domains.

Summary updated

Abstract

Retrieval over visually rich documents is essential for tasks such as legal discovery, scientific search, and enterprise knowledge management. Existing approaches fall into two paradigms: single-vector retrieval, which is efficient but coarse, and multi-vector retrieval, which is accurate but computationally expensive. To address this trade-off, we propose HEAVEN, a two-stage hybrid-vector framework. In the first stage, HEAVEN efficiently retrieves candidate pages using a single-vector method over Visually-Summarized Pages (VS-Pages), which assemble representative visual layouts from multiple pages. In the second stage, it reranks candidates with a multi-vector method while filtering query tokens by linguistic importance to reduce redundant computations. To evaluate retrieval systems under realistic conditions, we also introduce ViMDOC, the first benchmark for visually rich, multi-document, and long-document retrieval. Across four benchmarks, HEAVEN attains 99.87% of the Recall@1 performance of multi-vector models on average while reducing per-query computation by 99.82%, achieving efficiency and accuracy. Our code and datasets are available at: https://github.com/juyeonnn/HEAVEN
BibTeX
@inproceedings{kim2026hybrid,
  author = {Juyeon Kim and Geon Lee and Dongwon Choi and Taeuk Kim and Kijung Shin},
  title = {{Hybrid-Vector Retrieval for Visually Rich Documents: Combining Single-Vector Efficiency and Multi-Vector Accuracy}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.54},
}

Loading related content...

Figure