LMEG-Doc: Layout-Grounded Multi-View Evidence Graphs for Multimodal Long-Document Retrieval

Changcheng Fu, Xiao Liu, Qi Yu, Yage Wang, Derek Young, Giri IyengararXiv 2026 · arXiv

TL;DR

LMEG-Doc is a novel framework that treats long, multimodal documents as "evidence graphs" rather than a collection of isolated pages. By explicitly modeling the relationships between text, figures, and tables—even across distant pages—it significantly improves the retrieval of complementary information, boosting downstream question-answering accuracy by up to 15.8%.

What Problem Does This Paper Address?

In the world of Retrieval-Augmented Generation (RAG), we often treat documents as a series of independent "chunks." However, real-world documents—like financial reports or scientific papers—are deeply interconnected.

The authors identify a "fragmented evidence" problem: a user's question might perfectly match a figure caption (high lexical similarity), but the actual answer resides within the figure itself (which might have low direct similarity to the text-based query). Current retrievers often miss these "supporting" elements because they rank items in isolation. This is especially problematic in long documents where a paragraph on page 5 might refer to a table on page 12. LMEG-Doc aims to bridge these gaps by understanding document layout and structure.

Methods

The LMEG-Doc framework operates in three main stages:

  1. Evidence Graph Construction: The system parses documents into spatially grounded elements (headings, paragraphs, figures, tables). It then builds a graph where nodes are elements and edges represent structural relationships like reading order, caption-to-visual links, and spatial proximity.
  2. Cross-Page Relation Refinement (CRR): This is a key innovation. To find links across distant pages, the system uses "cues" (like shared numerical anchors or textual references) to propose potential connections. A Vision-Language Model (VLM) then verifies these pairs offline, adding high-confidence "cross-page" edges to the graph.
  3. Graph-Guided Retrieval: When a query comes in, LMEG-Doc calculates initial relevance scores using a "multi-view" approach (combining BM25, BGE-M3, SigLIP 2, and ColPali). It then uses Personalized PageRank (PPR) to propagate these scores through the graph. If a caption gets a high score, its associated table or figure receives a "relevance boost" via the graph edges.

Datasets

The authors evaluated LMEG-Doc on three challenging multimodal benchmarks:

  • MMDocIR: A large-scale benchmark with 1,658 questions across 10 domains, focusing on both page-level and layout-element-level retrieval.
  • DocScope: A dataset designed for verifiable reasoning in long documents.
  • FinRAGBench-V: A specialized benchmark focused on financial reports, which typically feature complex tables and long-range dependencies.

Key Insights and Ideas

  • Relationships > Individual Similarity: The core "aha moment" is that an element's utility is often defined by its context. By moving from a "rank-by-similarity" to a "rank-by-connectivity" approach, the retriever behaves more like a human researcher who follows references.
  • VLM as an Offline Architect: Instead of using expensive VLMs at query time, LMEG-Doc uses them during the indexing phase to "verify" the document's internal structure. This keeps the retrieval phase fast while benefiting from deep multimodal understanding.
  • Intra-Group Refinement: The paper introduces a strategy to ensure that "visual groups" (a figure and its explanation) are treated as a single unit of evidence, preventing the model from retrieving a figure without its necessary context.

Key Results

LMEG-Doc demonstrates impressive gains across the board without requiring task-specific fine-tuning:

  • Retrieval Performance: On MMDocIR, it achieved a 70.7% layout recall (L@10), outperforming existing retrievers that were specifically fine-tuned on that dataset.
  • Downstream QA Boost: When integrated into a QA pipeline with Qwen3.8-27B, LMEG-Doc improved answer accuracy by 7.6% on DocScope and a massive 15.8% on FinRAGBench-V.
  • Competitive Edge: The system outperformed several proprietary models, including Gemini 3.1 Pro and GPT-6.1 Sol (as cited in the paper), demonstrating that better retrieval can often compensate for smaller model sizes.

Limitations and Future Work

  • Parsing Dependency: The system's performance is inherently capped by the quality of the initial document parsing (e.g., Nemotron-Parse). Errors in bounding box detection or element classification will propagate through the graph.
  • Offline Overhead: While retrieval is fast, the initial graph construction—specifically the VLM-based verification of cross-page links—adds significant pre-processing time and computational cost.
  • Multi-Document Scaling: Currently, the framework focuses on intra-document relationships. A logical next step would be extending this to "cross-document" evidence graphs for even more complex research tasks.

Summary updated

Abstract

Retrieving relevant evidence from multimodal long documents remains challenging, as supporting information is often distributed across different pages and modalities. Existing retrieval methods primarily rank document elements by their individual similarity to a question, often overlooking relationships among complementary evidence. Consequently, relevant figures and tables may receive low retrieval scores even when they contain essential information. To address this limitation, we introduce LMEG-Doc, a layout-grounded multi-view evidence graph framework for multimodal long-document retrieval. We first construct an evidence graph that captures structural and contextual relationships among document elements, and introduce Cross-Page Relation Refinement (CRR) to discover additional connections across distant pages through VLM-based verification. Building on this graph, we combine multimodal retrieval signals with graph-guided relevance propagation to recover connected supporting evidence at both layout and page levels. Experiments on MMDocIR, DocScope, and FinRAGBench-V demonstrate consistent retrieval improvements, including outperforming retrievers fine-tuned on MMDocIR without task-specific training. Furthermore, augmenting existing question-answering pipelines with our retrieved evidence improves answer accuracy by 7.6% on DocScope and 15.8% on FinRAGBench-V, demonstrating the effectiveness of relationship-aware multimodal retrieval.
BibTeX
@misc{fu2026lmeg,
  author = {Changcheng Fu and Xiao Liu and Qi Yu and Yage Wang and Derek Young and Giri Iyengar},
  title = {{LMEG-Doc: Layout-Grounded Multi-View Evidence Graphs for Multimodal Long-Document Retrieval}},
  year = {2026},
  eprint = {2611.18901},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18901},
}
Figure