SRFUND: A Multi-Granularity Hierarchical Structure Reconstruction Benchmark in Form Understanding

Jiefeng Ma, Yan Wang, Chenyu Liu, Jun Du, Yu Hu, Zhenrong Zhang, Pengfei Hu, Qing Wang, Jianshu Zhang

TL;DR

SRFUND is a new multilingual benchmark that challenges AI models to go beyond simple text extraction and reconstruct the full hierarchical structure of complex forms. By providing refined annotations across eight languages, it shifts the focus from local "key-value" pairs to a global understanding of how words, lines, and entities build a complete document tree.

What Problem Does This Paper Address?

In the world of Document AI, "Form Understanding" is the art of turning a messy scan of an invoice or tax return into structured data. While we’ve made progress with datasets like FUNSD and XFUND, existing benchmarks are often too "local." They focus on identifying individual entities (like a "Total Amount" box) or simple links (connecting a "Name" label to a "John Doe" value).

However, real documents are hierarchical. Words form text lines, lines form entities, and entities are nested within sections or tables. Existing datasets often ignore these layers or fail to handle complex structures like multi-item tables. This "granularity gap" prevents models from truly understanding the global logic of a document.

Methods

To bridge this gap, researchers introduced SRFUND (Structured Reconstruction Form UNderstanding Dataset). Instead of starting from scratch, they meticulously "upgraded" the popular FUNSD and XFUND datasets to support multi-granularity analysis.

They defined five core tasks to evaluate a model's structural "IQ":

  1. Word to text-line merging: Grouping individual words into coherent lines.
  2. Text-line to entity merging: Combining lines into functional entities.
  3. Entity category classification: Labeling entities (Header, Question, Answer, Other).
  4. Item table localization: Identifying the boundaries of tables and their internal rows/headers.
  5. Entity-based hierarchical structure recovery: The "final boss" task—reconstructing the entire document's dependency tree.
Hierarchical document tree structure

To set the bar, the authors tested three types of architectures: vision-only (e.g., YOLOX), language-only (e.g., XLM-RoBERTa), and multi-modal models that see both pixels and text (e.g., LayoutXLM, GraphDoc). They also evaluated the latest LLMs, like GPT-4o, to see how they handle zero-shot form reconstruction.

Datasets

The benchmark consists of 1,592 form images spanning eight languages: English, Chinese, Japanese, German, French, Spanish, Italian, and Portuguese.

Compared to its predecessors, SRFUND is significantly denser:

  • It adds over 112,000 text-line annotations.
  • It introduces 15,362 new or modified entities.
  • It features an average "form tree depth" of 3.049, nearly double that of previous datasets, indicating much more complex logical nesting.

Key Insights and Ideas

  • The Necessity of Multi-modality: The results confirm that you can't understand a form by text alone. Vision-only models struggle with semantic continuity, while text-only models are "blind" to the spatial cues (like borders and alignment) that define a table.
  • Tables are the "Final Boss": Item table localization remains incredibly difficult. Even the best models struggled here (F1 scores around 0.30), proving that identifying nested rows and headers without explicit borders is a major open challenge.
Item table localization example
  • Cross-Lingual Structural Synergy: Interestingly, models trained on one language (like Spanish) performed surprisingly well on others (like German). This suggests that while the words change, the visual grammar of forms—how we use space to imply hierarchy—is somewhat universal.
  • Specialized Models > General LLMs: While GPT-4o showed reasonable zero-shot capabilities, it was consistently outperformed by smaller, domain-specific models like GraphDoc that are pre-trained specifically on document layouts.

Key Results

  • Hierarchical Recovery: The multi-modal GraphDoc model emerged as the leader, achieving an average F1 score of 0.7349 on the full-document structure recovery task.
  • Entity Classification: LayoutXLM dominated this category with an F1 of 0.9248, showing that pre-training on massive amounts of multilingual document data pays off for identifying "what" a box is.
  • LLM Performance: GPT-4o-mini actually outperformed the standard GPT-4o in the "Text-line to entity merging" task, likely because it was less prone to "hallucinating" new bounding boxes and stuck closer to the provided input coordinates.

Limitations and Future Work

The authors acknowledge that SRFUND is still relatively small (under 2,000 images) and consists primarily of grayscale scans. Real-world documents are often in color and feature much noisier backgrounds.

Future work will likely focus on:

  • Scaling up: Using automated pipelines to generate initial annotations for larger datasets like DocLayNet.
  • Complex Layouts: Moving beyond standard registration forms to more "wild" document types.
  • Table Parsing: Developing more robust architectures specifically for the "item table localization" task, which remains the weakest link in current document understanding pipelines.

Summary updated

Abstract

Accurately identifying and organizing textual content is crucial for the automation of document processing in the field of form understanding. Existing datasets, such as FUNSD and XFUND, support entity classification and relationship prediction tasks but are typically limited to local and entity-level annotations. This limitation overlooks the hierarchically structured representation of documents, constraining comprehensive understanding of complex forms. To address this issue, we present the SRFUND, a hierarchically structured multi-task form understanding benchmark. SRFUND provides refined annotations on top of the original FUNSD and XFUND datasets, encompassing five tasks: (1) word to text-line merging, (2) text-line to entity merging, (3) entity category classification, (4) item table localization, and (5) entity-based full-document hierarchical structure recovery. We meticulously supplemented the original dataset with missing annotations at various levels of granularity and added detailed annotations for multi-item table regions within the forms. Additionally, we introduce global hierarchical structure dependencies for entity relation prediction tasks, surpassing traditional local key-value associations. The SRFUND dataset includes eight languages including English, Chinese, Japanese, German, French, Spanish, Italian, and Portuguese, making it a powerful tool for cross-lingual form understanding. Extensive experimental results demonstrate that the SRFUND dataset presents new challenges and significant opportunities in handling diverse layouts and global hierarchical structures of forms, thus providing deep insights into the field of form understanding. The original dataset and implementations of baseline methods are available at https://sprateam-ustc.github.io/SRFUND.
BibTeX
@inproceedings{ma2024srfund,
  author = {Jiefeng Ma and Yan Wang and Chenyu Liu and Jun Du and Yu Hu and Zhenrong Zhang and Pengfei Hu and Qing Wang and Jianshu Zhang},
  title = {{SRFUND: A Multi-Granularity Hierarchical Structure Reconstruction Benchmark in Form Understanding}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure