Abstract
Accurately identifying and organizing textual content is crucial for the automation of document processing in the field of form understanding. Existing datasets, such as FUNSD and XFUND, support entity classification and relationship prediction tasks but are typically limited to local and entity-level annotations. This limitation overlooks the hierarchically structured representation of documents, constraining comprehensive understanding of complex forms. To address this issue, we present the SRFUND, a hierarchically structured multi-task form understanding benchmark. SRFUND provides refined annotations on top of the original FUNSD and XFUND datasets, encompassing five tasks: (1) word to text-line merging, (2) text-line to entity merging, (3) entity category classification, (4) item table localization, and (5) entity-based full-document hierarchical structure recovery. We meticulously supplemented the original dataset with missing annotations at various levels of granularity and added detailed annotations for multi-item table regions within the forms. Additionally, we introduce global hierarchical structure dependencies for entity relation prediction tasks, surpassing traditional local key-value associations. The SRFUND dataset includes eight languages, making it a powerful tool for cross-lingual form understanding.
Introduction
Automating the processing of forms—such as tax returns, invoices, and medical records—is a cornerstone of document AI. However, existing benchmarks like FUNSD and XFUND suffer from a "local" bias, focusing primarily on identifying individual entities or simple key-value pairs. Real-world forms are inherently hierarchical: words form lines, lines form semantic entities, and entities are organized into complex sections or tables. Current datasets lack the granularity to train models on this full reconstruction pipeline, leading to fragmented understanding where models fail to grasp a document's global logical flow.
The SRFUND Benchmark
To address these limitations, we introduce SRFUND, a benchmark designed to evaluate document structure reconstruction at multiple levels of granularity. The dataset encompasses 1,592 form images across eight languages (English, Chinese, Japanese, German, French, Spanish, Italian, and Portuguese). We significantly enhanced the original source data by adding over 15,000 new entity annotations and 75,000 new relationship links, increasing the average form tree depth from approximately 1.5 to 3.049.
SRFUND defines five distinct tasks:
- Word to text-line merging: Grouping individual words into coherent lines.
- Text-line to entity merging: Grouping lines into semantic entities.
- Entity category classification: Labeling entities (Header, Question, Answer, Other).
- Item table localization: Identifying complex multi-item table regions.
- Hierarchical structure recovery: Predicting global parent-child relationships between all entities.
Global Hierarchical Reconstruction
A key novelty of SRFUND is the shift from local key-value associations to a global tree structure. This requires models to understand the logical hierarchy of the entire document rather than relying on simple spatial proximity. Reconstructing this "big picture" remains a significant challenge for modern AI architectures.
Experimental Results and Insights
We established baselines using vision-only, language-only, and multi-modal models. Multi-modal models like LayoutXLM and GraphDoc, which integrate text, visual features, and 2D layout coordinates, consistently outperformed single-modality approaches.
- The Granularity Gap: Models often excel at classifying entities but struggle to correctly group the constituent words, highlighting a gap in low-level structural understanding.
- The Table Challenge: Item table localization proved to be the most difficult task. Because a table is only considered correct if every constituent entity is perfectly grouped, F1 scores for even top-tier models often fell below 0.30.
- Structure Recovery: GraphDoc led the hierarchical recovery task with an average F1 of 0.7349, indicating a ~26% performance gap remaining for perfect document reconstruction.
Conclusion
SRFUND reveals that while multi-modal models are proficient at local tasks, global hierarchical reconstruction and complex table localization remain open problems. Future research should focus on developing architectures that better align sentence-level semantics with hierarchical requirements and creating specialized modules for the grid-like logic of form tables. The dataset and baseline implementations are available at https://sprateam-ustc.github.io/SRFUND.
Summary updated