Visual Understanding of Complex Table Structures From Document Images

Sachin Raja, Ajoy Mondal, C.v. JawaharWACV 2022

TL;DR

This paper introduces a novel deep learning framework, TOD-Net and TSR-Net, for accurately recognizing the structure of complex tables in document images. By focusing on robust cell detection, including empty cells, and a new rectilinear adjacency formulation for structure recognition, they achieve state-of-the-art performance and introduce a new benchmark dataset, TUCD, to encourage further research.

What Problem Does This Paper Address?

Understanding tables within document images is crucial for extracting structured information, but it's a challenging task. Existing methods struggle with the high diversity of table layouts, varying cell alignments, and the presence of empty cells, which can carry implicit semantic meaning. Accurately identifying individual cells, especially empty ones, and understanding their relationships (like spanning multiple rows or columns) is key to extracting the table's underlying structure. This paper aims to address these challenges by proposing a more robust cell detection mechanism and a novel approach to structure recognition that better captures inherent table alignments.

Methods

The authors propose a two-stage approach:

  • Cell Detection (TOD-Net): They build upon the Mask R-CNN architecture, augmenting it with additional loss components. These include:
  • Continuity Loss: Encourages horizontally adjacent objects to end and start at the same x-coordinate, and vertically adjacent objects to do the same for y-coordinates.
  • Overlap Loss: Acts as an L2 regularizer to minimize overlapping regions between predicted table objects.
  • Channel Attention: Incorporates L1-regularized channel-wise attention to identify sparse patterns for accurate cell bounding box detection.
  • Trainable Loss Weights: A min-max optimization problem is formulated to dynamically adjust the weights of alignment, continuity, and overlap losses, allowing the model to adapt to different visual characteristics.
  • Structure Recognition (TSR-Net): This stage uses a graph-based formulation, specifically leveraging a DGCNN architecture. Instead of traditional row/column adjacency matrices, they employ rectilinear adjacency matrices (left, right, top, bottom). This approach aims to better capture long-range dependencies, especially in dense tables, and simplifies post-processing for identifying row/column spanning cells.

Datasets

  • Training: FinTabNet [41] was used for training the cell, row, and column detection components.
  • Evaluation: The authors evaluated their method on several existing datasets: ICDAR-2013 [8], CTDaR [7], SciTSR [4], PubTabNet [42], and FinTabNet [41].
  • New Benchmark: They introduce TUCD (Table Understanding for Complex Documents), a new dataset comprising 4500 manually annotated table images from business annual reports. TUCD is designed to have a high diversity of table layouts and features ground truth annotations that preserve native alignment constraints, including annotations for empty cells.

Key Insights and Ideas

  • Empty cells are semantically important: The paper strongly emphasizes that empty cells are not just blank spaces but can carry implicit meanings (e.g., zero values, "not applicable"). Their accurate detection is crucial for correct table understanding.
  • Human-like alignment is key: The authors argue that table cells, from a human perspective, adhere to natural alignment and continuity constraints. Their method aims to mimic this by incorporating losses that enforce these properties.
  • Rectilinear adjacencies simplify structure: Moving away from row/column adjacencies to rectilinear ones provides a more direct way to model relationships and reduces the need for complex post-processing heuristics to determine spanning cells.
  • Higher IoU thresholds are more informative: The paper advocates for evaluating cell detection at higher Intersection over Union (IoU) thresholds (e.g., 0.7 and above) to better assess a method's robustness and its ability to capture the full extent of cells, including empty ones.

Key Results

  • The proposed framework achieves state-of-the-art performance, improving the previous best by a 2.7% average F1-score on benchmark datasets.
  • Their method demonstrates improved F1-scores for cell detection, particularly at higher IoU thresholds.
  • The introduction of continuity and overlap losses led to improved cell detection performance.
  • The use of trainable loss weights allows for dynamic emphasis on different structural constraints, contributing to faster convergence and better performance.

Limitations and Future Work

  • Sparse tables with many empty cells: The authors acknowledge that their solution struggles with very sparse tables where most cells are empty.
  • Inference time for dense tables: For very dense tables with many cells, the inference time can be significant (up to 50 seconds for tables with 200 cells), due to the quadratic complexity of pairwise cell comparisons.
  • Future work: The authors suggest extending their trainable loss weights approach to other niche domains like graph/chart understanding and establishing reading order from document images. They also plan to address the limitations with sparse tables.

Summary updated

BibTeX
@inproceedings{raja2022visual,
  author = {Sachin Raja and Ajoy Mondal and C.V. Jawahar},
  title = {{Visual Understanding of Complex Table Structures From Document Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}

Loading related content...

Figure