Abstract
While Multi-modal Large Language Models (MLLMs) have demonstrated remarkable capabilities in general scene description, they often struggle with complex, text-rich documents. This paper introduces TGDoc, a novel framework designed to bridge the "spatial disconnect" in document understanding by incorporating text-grounding—the ability to explicitly locate text within an image using bounding boxes. By leveraging a specialized training pipeline and a new large-scale dataset of PowerPoint slides, TGDoc achieves state-of-the-art performance across multiple benchmarks, significantly reducing hallucinations and enhancing spatial reasoning.
Introduction
Current MLLMs like LLaVA and MiniGPT-4 frequently encounter difficulties when processing dense documents, such as PowerPoint slides or technical pages. While these models may recognize specific text strings, they often fail to understand the spatial layout or the relationship between different visual elements. This lack of spatial awareness leads to two primary failure modes: hallucinations (claiming text exists where it does not) and weak reasoning (failing to associate headers with their respective columns or data points). TGDoc addresses these issues by treating text-grounding as a fundamental component of the model's reasoning process.
The TGDoc Architecture
TGDoc utilizes a standard MLLM architecture consisting of a vision encoder (CLIP-ViT-L/14) and a large language model (Vicuna-7B), connected via a linear projection layer. The core innovation lies in the Grounded Representation format. During inference, the model is trained to output spatial coordinates in the format [xmin, ymin, xmax, ymax] alongside its textual responses. This requirement forces the model to provide a "visual rationale" for its answers, effectively acting as a form of visual chain-of-thought.
Training Methodology and Datasets
The researchers employed a two-stage training strategy to instill spatial awareness:
- Pre-training: The model was first exposed to a dataset of 99,000 PowerPoint slides sourced from SlideShare. PowerPoint slides are uniquely valuable for this task because they contain digitally rendered, structured text with diverse layouts, providing a clean environment for learning text-layout associations.
- Instruction Fine-tuning: The model was refined using 12,000 multi-turn grounded conversations. This dataset includes 11,000 book covers and 1,000 high-resolution images. Using GPT-4, the authors generated complex Q&A pairs that require the model to justify its answers by identifying the bounding boxes of relevant text.
Experimental Results
TGDoc-336 (operating at 336px resolution) established new benchmarks for performance in text-rich scenarios. On the TextVQA benchmark, it achieved 46.18% accuracy, surpassing previous state-of-the-art models like UniDoc and LLaVAR. Even more significant was the jump in Key Information Extraction (KIE) tasks; on the POIE benchmark, TGDoc reached 22.16%, a substantial improvement over previous models that typically scored below 7%.
Qualitative analysis confirms that TGDoc can accurately highlight specific regions—such as the "Open" sign on a storefront or the "British Airways" text on an aircraft—while simultaneously answering complex questions about those objects. This grounding capability significantly reduces the likelihood of the model misattributing text or hallucinating non-existent details.
Conclusion and Future Work
TGDoc demonstrates that explicit text-grounding is a powerful mechanism for improving document understanding in MLLMs. By forcing the model to "point" to the evidence for its claims, the researchers have created a more reliable and reasoning-capable system. Future research will focus on improving the precision of bounding boxes, which can currently be limited by the vision encoder's training on natural objects, and exploring higher-resolution processing to handle even denser document layouts.
Summary updated