TL;DR
This paper introduces the Intelligent Document Parsing (IDP) framework, an end-to-end system that leverages large multimodal language models (MLLMs) for document parsing. IDP decouples content parsing from layout grounding, allowing MLLMs to focus on extracting and organizing textual information while a dedicated module handles visual element localization, leading to significant improvements over existing methods.
What Problem Does This Paper Address?
In today's digital world, a vast amount of information is stored in pixel-based formats like images and scanned PDFs. Extracting structured data from these documents is crucial for efficient storage, processing, and downstream applications. Traditional methods often break down document parsing into a pipeline of separate tasks (e.g., layout analysis, reading order prediction), which can lead to incomplete semantic understanding and error propagation. While recent approaches use multimodal large language models (MLLMs) for end-to-end parsing, they often struggle with accurately grounding visual information or intertwine content extraction and layout localization, hindering performance. This paper aims to bridge this gap by proposing a novel framework that effectively utilizes MLLMs for document parsing while addressing the challenges of content understanding and visual grounding.
Methods
The core innovation of the IDP framework lies in its decoupled decoding mechanism. Instead of forcing a single model to handle both content parsing and layout grounding simultaneously, IDP separates these tasks.
- IDP Representation Format: The authors propose a new document representation format, akin to HTML, that integrates layout element information, reading order, and logical relationships. This format allows for richer representation of multi-modal elements (like images and charts) compared to simple Markdown.
- Decoupled Decoding:
- Content Parsing: An MLLM (specifically, Qwen2.5-VL-3B) is used for content parsing. It extracts textual content and generates placeholders for multi-modal elements. Crucially, it also generates "layout queries" that act as signals for the grounding module.
- Layout Grounding: A separate spatial decoder (DETR-like) takes these layout queries and predicts the bounding boxes for the identified layout elements. This spatial decoder is activated only when the LLM has finished parsing a layout element, allowing it to focus solely on localization.
This decoupling allows the MLLM to concentrate on understanding and organizing the document's textual content, while the spatial decoder excels at precisely locating visual elements. The framework is trained in a three-stage process: multi-modal training for vision-language alignment, joint training of both decoders, and finally, fine-tuning the spatial decoder to enhance grounding capabilities.
Datasets
The paper evaluates IDP on several datasets:
- HRDoc-Simple (HRDS) and HRDoc-Hard (HRDH): These datasets contain documents from ACL and arXiv, respectively, used for training and evaluation.
- DocHieNet: A dataset with diverse document types in English and Chinese, used for evaluating hierarchical parsing.
- OmniDocBench: A benchmark for zero-shot evaluation on a comprehensive set of multi-type documents.
The authors also mention using a large corpus of annotated document parsing data for the initial multi-modal training stage.
Key Insights and Ideas
- Decoupling is Key: The most significant insight is that forcing MLLMs to perform both content parsing and layout grounding simultaneously leads to suboptimal performance. Separating these tasks allows each component to specialize and leverage its strengths, unlocking the full potential of MLLMs for document parsing.
- MLLMs for Content, Dedicated Module for Grounding: The IDP framework effectively uses MLLMs for their strength in understanding and generating structured text (like the HTML-like format) while relying on a specialized spatial decoder for accurate visual localization.
- HTML-like Representation Matters: Moving beyond simple Markdown for document representation is crucial for capturing multi-modal information, which is essential for comprehensive document understanding.
- Three-Stage Training is Beneficial: The proposed training strategy, moving from general multi-modal alignment to joint decoding and then spatial decoder fine-tuning, is critical for achieving high performance, especially in layout grounding.
Key Results
The IDP model demonstrates superior performance across various datasets compared to both open-source and closed-source baseline MLLMs, as well as pipeline-based methods.
- Grounding Performance: IDP significantly outperforms other MLLMs in layout grounding (G-F1 scores). For example, on HRDoc-Hard, IDP achieves a G-F1 of 96.11, while Qwen2.5-VL-7B (SFT) gets 90.32.
- Overall Performance: IDP achieves state-of-the-art results on HRDoc-Simple and HRDoc-Hard datasets, outperforming even larger closed-source models like Qwen-VL-Max.
- Zero-Shot Capability: IDP shows strong zero-shot performance on the OmniDocBench dataset, outperforming MLLM-based models and significantly outperforming pipeline methods that degrade in this scenario.
- Qualitative Improvements: Visualizations show that IDP produces more accurate and less hallucinated grounding boxes compared to Qwen-VL, which exhibits issues like shifted boxes and misplaced elements.
Limitations and Future Work
- Inter-page Relationships: The current IDP model does not explicitly consider relationships across multiple pages in a document. Since real-world scenarios often involve multi-page documents, extending IDP to support joint inference across pages is a promising future direction.
- Content Parsing Dependency: While IDP excels at layout grounding, the quality of content parsing by the MLLM can still influence the overall results. Further improvements in MLLM's content understanding capabilities would directly benefit IDP.
Summary updated