SRFUND: A Multi-Granularity Hierarchical Structure Reconstruction Benchmark in Form Understanding

Jiefeng Ma, Yan Wang, Chenyu Liu, Jun Du, Yu Hu, Zhenrong Zhang, Pengfei Hu, Qing Wang, Jianshu Zhang

Abstract

Accurately identifying and organizing textual content is crucial for the automation of document processing in the field of form understanding. Existing datasets, such as FUNSD and XFUND, support entity classification and relationship prediction tasks but are typically limited to local and entity-level annotations. This limitation overlooks the hierarchically structured representation of documents, constraining comprehensive understanding of complex forms. To address this issue, we present the SRFUND, a hierarchically structured multi-task form understanding benchmark. SRFUND provides refined annotations on top of the original FUNSD and XFUND datasets, encompassing five tasks: (1) word to text-line merging, (2) text-line to entity merging, (3) entity category classification, (4) item table localization, and (5) entity-based full-document hierarchical structure recovery. We meticulously supplemented the original dataset with missing annotations at various levels of granularity and added detailed annotations for multi-item table regions within the forms. Additionally, we introduce global hierarchical structure dependencies for entity relation prediction tasks, surpassing traditional local key-value associations. The SRFUND dataset includes eight languages, making it a powerful tool for cross-lingual form understanding.

Introduction

Automating the processing of forms—such as tax returns, invoices, and medical records—is a cornerstone of document AI. However, existing benchmarks like FUNSD and XFUND suffer from a "local" bias, focusing primarily on identifying individual entities or simple key-value pairs. Real-world forms are inherently hierarchical: words form lines, lines form semantic entities, and entities are organized into complex sections or tables. Current datasets lack the granularity to train models on this full reconstruction pipeline, leading to fragmented understanding where models fail to grasp a document's global logical flow.

The SRFUND Benchmark

To address these limitations, we introduce SRFUND, a benchmark designed to evaluate document structure reconstruction at multiple levels of granularity. The dataset encompasses 1,592 form images across eight languages (English, Chinese, Japanese, German, French, Spanish, Italian, and Portuguese). We significantly enhanced the original source data by adding over 15,000 new entity annotations and 75,000 new relationship links, increasing the average form tree depth from approximately 1.5 to 3.049.

Multi-granularity annotations in SRFUND, illustrating the progression from word-level detection to complex table localization.

SRFUND defines five distinct tasks:

  1. Word to text-line merging: Grouping individual words into coherent lines.
  2. Text-line to entity merging: Grouping lines into semantic entities.
  3. Entity category classification: Labeling entities (Header, Question, Answer, Other).
  4. Item table localization: Identifying complex multi-item table regions.
  5. Hierarchical structure recovery: Predicting global parent-child relationships between all entities.

Global Hierarchical Reconstruction

A key novelty of SRFUND is the shift from local key-value associations to a global tree structure. This requires models to understand the logical hierarchy of the entire document rather than relying on simple spatial proximity. Reconstructing this "big picture" remains a significant challenge for modern AI architectures.

A visualization of the global hierarchical tree structure recovered from a form, showing the complex parent-child relationships between entities.

Experimental Results and Insights

We established baselines using vision-only, language-only, and multi-modal models. Multi-modal models like LayoutXLM and GraphDoc, which integrate text, visual features, and 2D layout coordinates, consistently outperformed single-modality approaches.

  • The Granularity Gap: Models often excel at classifying entities but struggle to correctly group the constituent words, highlighting a gap in low-level structural understanding.
  • The Table Challenge: Item table localization proved to be the most difficult task. Because a table is only considered correct if every constituent entity is perfectly grouped, F1 scores for even top-tier models often fell below 0.30.
  • Structure Recovery: GraphDoc led the hierarchical recovery task with an average F1 of 0.7349, indicating a ~26% performance gap remaining for perfect document reconstruction.

Conclusion

SRFUND reveals that while multi-modal models are proficient at local tasks, global hierarchical reconstruction and complex table localization remain open problems. Future research should focus on developing architectures that better align sentence-level semantics with hierarchical requirements and creating specialized modules for the grid-like logic of form tables. The dataset and baseline implementations are available at https://sprateam-ustc.github.io/SRFUND.

Summary updated

Abstract

Accurately identifying and organizing textual content is crucial for the automation of document processing in the field of form understanding. Existing datasets, such as FUNSD and XFUND, support entity classification and relationship prediction tasks but are typically limited to local and entity-level annotations. This limitation overlooks the hierarchically structured representation of documents, constraining comprehensive understanding of complex forms. To address this issue, we present the SRFUND, a hierarchically structured multi-task form understanding benchmark. SRFUND provides refined annotations on top of the original FUNSD and XFUND datasets, encompassing five tasks: (1) word to text-line merging, (2) text-line to entity merging, (3) entity category classification, (4) item table localization, and (5) entity-based full-document hierarchical structure recovery. We meticulously supplemented the original dataset with missing annotations at various levels of granularity and added detailed annotations for multi-item table regions within the forms. Additionally, we introduce global hierarchical structure dependencies for entity relation prediction tasks, surpassing traditional local key-value associations. The SRFUND dataset includes eight languages including English, Chinese, Japanese, German, French, Spanish, Italian, and Portuguese, making it a powerful tool for cross-lingual form understanding. Extensive experimental results demonstrate that the SRFUND dataset presents new challenges and significant opportunities in handling diverse layouts and global hierarchical structures of forms, thus providing deep insights into the field of form understanding. The original dataset and implementations of baseline methods are available at https://sprateam-ustc.github.io/SRFUND
BibTeX
@inproceedings{ma2024srfund,
  author = {Jiefeng Ma and Yan Wang and Chenyu Liu and Jun Du and Yu Hu and Zhenrong Zhang and Pengfei Hu and Qing Wang and Jianshu Zhang},
  title = {{SRFUND: A Multi-Granularity Hierarchical Structure Reconstruction Benchmark in Form Understanding}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure