TL;DR
This paper introduces a novel "cell-based layout" representation for document understanding, which models component positions using row and column indices, mimicking human reading habits. This approach, combined with a multi-scale representation and data augmentation, significantly improves performance on named entity recognition and document classification tasks, achieving state-of-the-art results on several benchmarks.
What Problem Does This Paper Address?
Document understanding, the task of extracting structured information from documents like forms and receipts, is challenging due to its cross-modal nature (text, visuals, layout). While text and visual information extraction have advanced significantly, effectively utilizing layout information has lagged. Existing methods often rely on absolute or relative coordinates, which can be sensitive to OCR inaccuracies and struggle to capture the latent, hierarchical relationships inherent in document layouts. This makes it difficult for models to learn how components are organized, hindering their ability to understand complex documents.
Methods
The core innovation is the cell-based layout representation. Instead of using raw coordinates, this method assigns a row and column index to each component (e.g., text block, image) based on its position within the document. This is achieved by sorting the coordinates of component bounding boxes and defining cells based on these sorted values. This approach is more aligned with how humans perceive document structure, making it easier for models to learn relationships like components being in the same row or column.
To further capture document complexity, the authors propose a multi-scale layout. Documents have a natural hierarchy (e.g., document -> sections -> sentences -> words). The multi-scale layout feeds both word-level and token-level cell information into the model. This allows the model to learn from different granularities of document structure without requiring architectural changes or additional annotations.
Finally, a data augmentation technique is introduced to simulate real-world variations like handwritten text and camera motion. This involves randomly resizing bounding box patches, which diversifies the training data without altering the underlying cell-based layout structure.
Datasets
The proposed method was evaluated on the following datasets:
- FUNSD (Form Understanding in Noisy Scanned Documents): A dataset for form understanding with annotations for semantic entities like "question," "answer," "header," and "other."
- CORD (Consolidated Receipt Dataset): A dataset for receipt key information extraction, featuring 30 semantic labels across 4 categories.
- RVL-CDIP (Rider-Visual-Language Corpus of Document Images): A large-scale dataset for document classification, containing 16 categories.
Key Insights and Ideas
- Human-like spatial representation: The cell-based layout, using row/column indices, is more intuitive and directly reflects human reading habits than raw coordinates, leading to better model understanding of spatial relationships.
- Hierarchical structure matters: Explicitly modeling the multi-scale nature of documents (word-level and token-level) allows models to capture richer hierarchical layout information, improving performance.
- Simplicity and reusability: The cell-based layout can be integrated into existing models (like LayoutLMv2/v3) without significant architectural changes, making it broadly applicable.
- Data augmentation for robustness: The proposed augmentation method enhances robustness by simulating real-world variations without disrupting the learned layout structure.
Key Results
The proposed multi-scale cell-based layout achieved state-of-the-art (SOTA) performance on several tasks:
- Named Entity Recognition (NER):
- On the FUNSD dataset, the method improved the F1 score by up to 5.08% without pre-training and achieved 93.76% with pre-training, outperforming even larger baseline models.
- On the CORD dataset, the multi-scale cell-based layout achieved an F1 score of 97.23% (with pre-training), a significant improvement of 16.41% over the baseline.
- Document Classification:
- On the RVL-CDIP dataset, the cell-based layout improved accuracy by 7.5% with only 0.1% of the training data and reached 78.69% accuracy with 10% of the data, outperforming the baseline.
Limitations and Future Work
- Data Augmentation Nuances: The paper notes that updating bounding boxes during data augmentation can lead to better performance, suggesting that the model benefits from learning with slightly altered spatial information that still respects the cell-based structure.
- Pre-trained Model Improvement: While the cell-based layout improves performance, the gains are sometimes smaller on already pre-trained models, as these models have already learned significant visual and textual features. The authors suggest this is reasonable, as the cell-based layout is a new input modality not seen during pre-training.
- Future Directions: The authors believe their cell-based layout can be easily adapted to other tasks requiring spatial information, such as image detection. They also suggest further exploration of multi-scale representations and their integration with various backbone models.
Summary updated