TL;DR
LMEG-Doc is a novel framework that treats long, multimodal documents as "evidence graphs" rather than a collection of isolated pages. By explicitly modeling the relationships between text, figures, and tables—even across distant pages—it significantly improves the retrieval of complementary information, boosting downstream question-answering accuracy by up to 15.8%.
What Problem Does This Paper Address?
In the world of Retrieval-Augmented Generation (RAG), we often treat documents as a series of independent "chunks." However, real-world documents—like financial reports or scientific papers—are deeply interconnected.
The authors identify a "fragmented evidence" problem: a user's question might perfectly match a figure caption (high lexical similarity), but the actual answer resides within the figure itself (which might have low direct similarity to the text-based query). Current retrievers often miss these "supporting" elements because they rank items in isolation. This is especially problematic in long documents where a paragraph on page 5 might refer to a table on page 12. LMEG-Doc aims to bridge these gaps by understanding document layout and structure.
Methods
The LMEG-Doc framework operates in three main stages:
- Evidence Graph Construction: The system parses documents into spatially grounded elements (headings, paragraphs, figures, tables). It then builds a graph where nodes are elements and edges represent structural relationships like reading order, caption-to-visual links, and spatial proximity.
- Cross-Page Relation Refinement (CRR): This is a key innovation. To find links across distant pages, the system uses "cues" (like shared numerical anchors or textual references) to propose potential connections. A Vision-Language Model (VLM) then verifies these pairs offline, adding high-confidence "cross-page" edges to the graph.
- Graph-Guided Retrieval: When a query comes in, LMEG-Doc calculates initial relevance scores using a "multi-view" approach (combining BM25, BGE-M3, SigLIP 2, and ColPali). It then uses Personalized PageRank (PPR) to propagate these scores through the graph. If a caption gets a high score, its associated table or figure receives a "relevance boost" via the graph edges.
Datasets
The authors evaluated LMEG-Doc on three challenging multimodal benchmarks:
- MMDocIR: A large-scale benchmark with 1,658 questions across 10 domains, focusing on both page-level and layout-element-level retrieval.
- DocScope: A dataset designed for verifiable reasoning in long documents.
- FinRAGBench-V: A specialized benchmark focused on financial reports, which typically feature complex tables and long-range dependencies.
Key Insights and Ideas
- Relationships > Individual Similarity: The core "aha moment" is that an element's utility is often defined by its context. By moving from a "rank-by-similarity" to a "rank-by-connectivity" approach, the retriever behaves more like a human researcher who follows references.
- VLM as an Offline Architect: Instead of using expensive VLMs at query time, LMEG-Doc uses them during the indexing phase to "verify" the document's internal structure. This keeps the retrieval phase fast while benefiting from deep multimodal understanding.
- Intra-Group Refinement: The paper introduces a strategy to ensure that "visual groups" (a figure and its explanation) are treated as a single unit of evidence, preventing the model from retrieving a figure without its necessary context.
Key Results
LMEG-Doc demonstrates impressive gains across the board without requiring task-specific fine-tuning:
- Retrieval Performance: On MMDocIR, it achieved a 70.7% layout recall (L@10), outperforming existing retrievers that were specifically fine-tuned on that dataset.
- Downstream QA Boost: When integrated into a QA pipeline with Qwen3.8-27B, LMEG-Doc improved answer accuracy by 7.6% on DocScope and a massive 15.8% on FinRAGBench-V.
- Competitive Edge: The system outperformed several proprietary models, including Gemini 3.1 Pro and GPT-6.1 Sol (as cited in the paper), demonstrating that better retrieval can often compensate for smaller model sizes.
Limitations and Future Work
- Parsing Dependency: The system's performance is inherently capped by the quality of the initial document parsing (e.g., Nemotron-Parse). Errors in bounding box detection or element classification will propagate through the graph.
- Offline Overhead: While retrieval is fast, the initial graph construction—specifically the VLM-based verification of cross-page links—adds significant pre-processing time and computational cost.
- Multi-Document Scaling: Currently, the framework focuses on intra-document relationships. A logical next step would be extending this to "cross-document" evidence graphs for even more complex research tasks.
Summary updated