TL;DR
While modern Multimodal Large Language Models (MLLMs) can answer questions about document images, they struggle to point to the exact visual text regions that justify their answers. This paper introduces TRIG (Text-Rich Image Grounding), a new benchmark and training suite evaluating whether MLLMs can produce supportive bounding boxes alongside their answers. The authors also propose two baseline approaches—an instruction-tuned model and a fast embedding-based patch retriever—that dramatically outperform existing models.
What Problem Does This Paper Address?
When an AI assistant answers a question about a scanned invoice, dense infographic, or financial chart, verifying its answer requires users to scour the document themselves. Grounding the answer in specific image regions builds user trust and curbs hallucinations.
However, existing visual grounding research predominantly centers on natural images (e.g., localizing "the red car on the left"). In contrast, document and infographic images present dense text, complex tabular structures, and intricate layouts. Prior document models either focus on raw text transcription (OCR/spotting) or document-level QA without providing answer-justifying bounding boxes.
Methods
The authors formalize the TRIG task under multiple evaluation settings and propose two baseline methodologies:
- Evaluation Protocols:
- OCR-free Grounding (Setting 1): The model receives only the image and question, and must output the answer plus exact bounding box coordinates from scratch (evaluated via pixel-level IoU).
- OCR-based Grounding (Setting 2): OCR bounding boxes and transcriptions are included in the prompt; the model selects which box indices support the answer (evaluated via instance-level IoU, Precision, Recall, and F1).
- OCR-boxes Only (Setting 3): Box coordinates are provided without text content to isolate visual/spatial reasoning from linguistic shortcuts.
- Proposed Baseline Approaches:
- Instruction-Tuned Baseline: Fine-tunes LLaVA-v1.5-13B on structured question-answer-box sequences via autoregressive next-token prediction.
- Embedding-Based Baseline (ColPali-inspired): Uses PaliGemma-3B as an encoder trained with contrastive InfoCE loss over patch-token similarity matrices. At inference time, it matches the concatenated question-and-answer text to visual patch embeddings using:
- Adjacent Patch Embedding Merging: A $3 \times 3$ spatial smoothing window over image patch embeddings.
- Two-Level Patch Selection: Identifies the top-5 candidate patches and progressively expands outward to neighboring high-scoring patches, avoiding disconnected false positives.
Datasets
The authors built their pipeline over four diverse text-rich sources: DocVQA (document forms/invoices), ChartQA (data plots), InfographicVQA (complex graphic layouts), and TRINS (posters and book covers).
- TRIG-Bench (Evaluation): 800 manually verified QA pairs (200 per source dataset) with human-inspected ground-truth bounding boxes.
- TRIG Training Set: Approximately 90k synthetic instruction instances (~22.4k ChartQA, ~29.1k DocVQA, ~19.8k InfographicsVQA, ~24.9k TRINS) created via an automated pipeline: PaddleOCR detects text $\rightarrow$ visual boxes with indexed labels are overlaid onto the image and fed to GPT-4o $\rightarrow$ a secondary LLM reflection/rectification step verifies the selection.
Key Insights and Ideas
- Open-source MLLMs fail on format, not just vision: Many open-source models scored near 0% IoU because they stubbornly answered the question and completely ignored the instruction to output coordinates (often having instruction-following rates below 30%).
- GPT-4o understands instructions, but lacks fine-grained document geometry: GPT-4o followed the formatting instructions 96–98% of the time. When provided OCR candidate boxes (Setting 2), its F1 jumped to 85.34%. Yet in the OCR-free setting (Setting 1), it achieved only 5.26% pixel IoU, proving that raw coordinate prediction from scratch remains fundamentally difficult even for frontier proprietary models.
- Visual index overlay resolves prompt misalignment: Simply providing OCR bounding coordinates in text causes severe spatial misalignment in LLMs. Burning visual box IDs directly onto the input image allowed GPT-4o to reliably correlate text descriptions with image regions during dataset generation.
Key Results
- OCR-Free Grounding (Setting 1):
- Standard open-source MLLMs (LLaVA-v1.6, Qwen2-VL, InternVL2) scored 0.00% to 1.86% average pixel IoU.
- GPT-4o scored 5.26%.
- The proposed Embedding-based model reached 11.82%, while the Instruction-tuned LLaVA reached 29.98%.
- OCR-Based Grounding (Setting 2):
- Most open-source MLLMs remained below 30% average score due to low instruction adherence.
- GPT-4o achieved 85.34%, while the authors' instruction model reached 69.77% and the embedding model achieved 52.84%.
- Inference Speed: The embedding-based method required only two forward passes (one image, one text), taking 0.74 seconds per document on an Nvidia A100 GPU versus 2.37 seconds for autoregressive token generation.
Limitations and Future Work
- Performance Trade-Offs: While the embedding-based retriever is over $3\times$ faster and bypasses instruction-following errors, its localization accuracy still lags behind autoregressive instruction tuning.
- Dependency on OCR Quality: For OCR-based pipelines, errors or omissions in the upstream OCR engine directly bottleneck grounding precision.
- Resolution and Granularity: Fixed-patch architectures (e.g., $32 \times 32$ grid tokens) struggle with sub-word or tiny text snippets in ultra-dense documents without heavy upsampling.
Summary updated