Towards Visual Text Grounding of Multimodal Large Language Model

Ming Li, Ruiyi Zhang, Chenguang Wang, Jian Chen, Jiuxiang Gu, Yufan Zhou, Franck Dernoncourt, Wanrong Zhu, Tianyi Zhou, Tong SunSubmission to ICLR 2026 · Rejected

TL;DR

While modern Multimodal Large Language Models (MLLMs) can answer questions about document images, they struggle to point to the exact visual text regions that justify their answers. This paper introduces TRIG (Text-Rich Image Grounding), a new benchmark and training suite evaluating whether MLLMs can produce supportive bounding boxes alongside their answers. The authors also propose two baseline approaches—an instruction-tuned model and a fast embedding-based patch retriever—that dramatically outperform existing models.

What Problem Does This Paper Address?

When an AI assistant answers a question about a scanned invoice, dense infographic, or financial chart, verifying its answer requires users to scour the document themselves. Grounding the answer in specific image regions builds user trust and curbs hallucinations.

However, existing visual grounding research predominantly centers on natural images (e.g., localizing "the red car on the left"). In contrast, document and infographic images present dense text, complex tabular structures, and intricate layouts. Prior document models either focus on raw text transcription (OCR/spotting) or document-level QA without providing answer-justifying bounding boxes.

Methods

The authors formalize the TRIG task under multiple evaluation settings and propose two baseline methodologies:

  1. Evaluation Protocols:
  2. OCR-free Grounding (Setting 1): The model receives only the image and question, and must output the answer plus exact bounding box coordinates from scratch (evaluated via pixel-level IoU).
  3. OCR-based Grounding (Setting 2): OCR bounding boxes and transcriptions are included in the prompt; the model selects which box indices support the answer (evaluated via instance-level IoU, Precision, Recall, and F1).
  4. OCR-boxes Only (Setting 3): Box coordinates are provided without text content to isolate visual/spatial reasoning from linguistic shortcuts.
  5. Proposed Baseline Approaches:
  6. Instruction-Tuned Baseline: Fine-tunes LLaVA-v1.5-13B on structured question-answer-box sequences via autoregressive next-token prediction.
  7. Embedding-Based Baseline (ColPali-inspired): Uses PaliGemma-3B as an encoder trained with contrastive InfoCE loss over patch-token similarity matrices. At inference time, it matches the concatenated question-and-answer text to visual patch embeddings using:
  8. Adjacent Patch Embedding Merging: A $3 \times 3$ spatial smoothing window over image patch embeddings.
  9. Two-Level Patch Selection: Identifies the top-5 candidate patches and progressively expands outward to neighboring high-scoring patches, avoiding disconnected false positives.

Datasets

The authors built their pipeline over four diverse text-rich sources: DocVQA (document forms/invoices), ChartQA (data plots), InfographicVQA (complex graphic layouts), and TRINS (posters and book covers).

  • TRIG-Bench (Evaluation): 800 manually verified QA pairs (200 per source dataset) with human-inspected ground-truth bounding boxes.
  • TRIG Training Set: Approximately 90k synthetic instruction instances (~22.4k ChartQA, ~29.1k DocVQA, ~19.8k InfographicsVQA, ~24.9k TRINS) created via an automated pipeline: PaddleOCR detects text $\rightarrow$ visual boxes with indexed labels are overlaid onto the image and fed to GPT-4o $\rightarrow$ a secondary LLM reflection/rectification step verifies the selection.

Key Insights and Ideas

  • Open-source MLLMs fail on format, not just vision: Many open-source models scored near 0% IoU because they stubbornly answered the question and completely ignored the instruction to output coordinates (often having instruction-following rates below 30%).
  • GPT-4o understands instructions, but lacks fine-grained document geometry: GPT-4o followed the formatting instructions 96–98% of the time. When provided OCR candidate boxes (Setting 2), its F1 jumped to 85.34%. Yet in the OCR-free setting (Setting 1), it achieved only 5.26% pixel IoU, proving that raw coordinate prediction from scratch remains fundamentally difficult even for frontier proprietary models.
  • Visual index overlay resolves prompt misalignment: Simply providing OCR bounding coordinates in text causes severe spatial misalignment in LLMs. Burning visual box IDs directly onto the input image allowed GPT-4o to reliably correlate text descriptions with image regions during dataset generation.

Key Results

  • OCR-Free Grounding (Setting 1):
  • Standard open-source MLLMs (LLaVA-v1.6, Qwen2-VL, InternVL2) scored 0.00% to 1.86% average pixel IoU.
  • GPT-4o scored 5.26%.
  • The proposed Embedding-based model reached 11.82%, while the Instruction-tuned LLaVA reached 29.98%.
  • OCR-Based Grounding (Setting 2):
  • Most open-source MLLMs remained below 30% average score due to low instruction adherence.
  • GPT-4o achieved 85.34%, while the authors' instruction model reached 69.77% and the embedding model achieved 52.84%.
  • Inference Speed: The embedding-based method required only two forward passes (one image, one text), taking 0.74 seconds per document on an Nvidia A100 GPU versus 2.37 seconds for autoregressive token generation.

Limitations and Future Work

  • Performance Trade-Offs: While the embedding-based retriever is over $3\times$ faster and bypasses instruction-following errors, its localization accuracy still lags behind autoregressive instruction tuning.
  • Dependency on OCR Quality: For OCR-based pipelines, errors or omissions in the upstream OCR engine directly bottleneck grounding precision.
  • Resolution and Granularity: Fixed-patch architectures (e.g., $32 \times 32$ grid tokens) struggle with sub-word or tiny text snippets in ultra-dense documents without heavy upsampling.

Summary updated

Abstract

Despite the existing evolution of Multimodal Large Language Models (MLLMs), a non-neglectable limitation remains in their struggle with visual text grounding, especially in text-rich images of documents. Document images, such as scanned forms and infographics, highlight critical challenges due to their complex layouts and textual content. However, current benchmarks do not fully address these challenges, as they mostly focus on visual grounding on natural images, rather than text-rich document images. Thus, to bridge this gap, we introduce TRIG, a novel task with a newly designed instruction dataset for benchmarking and improving the Text-Rich Image Grounding capabilities of MLLMs in document question-answering. Specifically, we propose an OCR-LLM-human interaction pipeline to create 800 manually annotated question-answer pairs as a benchmark and a large-scale training set of k synthetic data based on four diverse datasets. A comprehensive evaluation of various MLLMs on our proposed benchmark exposes substantial limitations in their grounding capability on text-rich images. In addition, we propose two simple and effective TRIG methods based on general instruction tuning and plug-and-play efficient embedding, respectively. By finetuning MLLMs on our synthetic dataset, they promisingly improve spatial reasoning and grounding capabilities.
BibTeX
@misc{li2025towards,
  author = {Ming Li and Ruiyi Zhang and Chenguang Wang and Jian Chen and Jiuxiang Gu and Yufan Zhou and Franck Dernoncourt and Wanrong Zhu and Tianyi Zhou and Tong Sun},
  title = {{Towards Visual Text Grounding of Multimodal Large Language Model}},
  year = {2025},
  eprint = {2504.04974},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.04974},
}
Figure