An open approach towards the benchmarking of table structure recognition systems

Asif Shahab, Faisal Shafait, Thomas Kieninger, Andreas DengelDocument Analysis Systems 2010 · DOI

TL;DR

This paper introduces an open framework for benchmarking table structure recognition systems, addressing the lack of standardized evaluation methods. They propose a novel color-coded image representation for ground truth and apply established image segmentation metrics, making their tools and datasets publicly available to foster reproducible research.

What Problem Does This Paper Address?

Recognizing and understanding the structure of tables within documents is a crucial but challenging task in document analysis. While many systems exist, there's a significant lack of standardized benchmarks and evaluation metrics, making it difficult to objectively compare their performance. Different systems excel in different scenarios, and without a common ground, researchers and developers struggle to determine the best approach for their specific needs or to track progress in the field. This paper aims to bridge this gap by providing an open, reproducible, and comprehensive benchmarking environment.

Methods

The core of the paper's contribution lies in establishing an open benchmarking framework. Key methods include:

  • Color-Coded Image Representation for Ground Truth: Instead of traditional bounding boxes, they propose encoding table structure (cells, rows, columns, and spanning cells) directly into an image's color channels (RGB). This allows for a pixel-accurate, hierarchical representation that can be visualized and processed using standard image segmentation evaluation techniques.
  • Application of Image Segmentation Metrics: They adapt established metrics from the image segmentation community (e.g., Correct Detections, Partial Detections, Over-Segmentations, Under-Segmentations, Missed Segments, False Positives) to evaluate table recognition performance at various levels of abstraction (table, row, column, cell, row/column spans).
  • "T-Truth" Ground Truth Preparation Tool: A user-friendly tool is developed to facilitate the manual creation of ground truth data. It allows users to draw table zones, mark row/column separators, and define spanning cells through intuitive mouse interactions.
  • Open Access: The authors commit to making their ground-truthing tool, the prepared ground truth data, and the evaluation framework publicly available to the research community.

Datasets

The authors utilized the following publicly available datasets for their experiments and benchmark preparation:

  • UNLV dataset: Contains 2889 scanned document images. For this work, they focused on 427 binary 300-dpi images with marked table zones.
  • UW-3 dataset: Consists of 1600 skew-corrected English document images. They selected approximately 120 images containing marked table zones.

The paper also highlights that they generated ground truth for these datasets using their T-Truth tool and made it publicly available.

Key Insights and Ideas

  • Bridging the Gap in Table Recognition Evaluation: The most significant insight is the realization that a lack of standardized evaluation is a major bottleneck in table recognition research. By providing an open framework, they aim to accelerate progress.
  • Image Segmentation as a Powerful Analogy: The paper effectively draws a parallel between table structure extraction and hierarchical image segmentation, enabling the reuse of well-established evaluation methodologies.
  • Color-Coding for Richer Ground Truth: Encoding hierarchical information into color channels offers a more expressive and flexible way to represent table structures compared to simple bounding boxes, especially for complex layouts with spanning cells.
  • The Importance of Openness: The commitment to releasing tools and data is crucial for fostering reproducible research and enabling direct comparisons between different systems.

Key Results

The paper presents evaluation results for the T-Recs system on the UNLV and UW-3 datasets. Key quantitative findings include:

  • T-Recs Performance on UW-3: For table detection, T-Recs achieved 10.71% correct detections. Cell detection was stronger, with 62.92% correct detections. Row and column span detection showed lower accuracy (0% and 32.35% respectively).
  • T-Recs Performance on UNLV: On the UNLV dataset, T-Recs showed higher table detection accuracy (34.86%) and cell detection accuracy (74.28%). Row and column span detection also saw improvements (0.84% and 15.24% respectively).
  • Performance on Multi-Column Layouts: Table 3 shows that T-Recs' performance varies significantly with the number of columns. It performs best on single-column layouts (50.40% correct table detection) and drops considerably for two- and three-column layouts.

Qualitatively, the paper demonstrates how their evaluation framework can visualize segmentation errors (false positives, under-segmentation, over-segmentation, missed segments) on the document images, providing clear insights into system weaknesses.

Limitations and Future Work

  • T-Recs Limitations: The authors acknowledge that T-Recs, being designed for single-column layouts, struggles with multi-column documents, which is reflected in the evaluation results. The system also relies on a set of numerical parameters that typically require manual tuning for different datasets.
  • Future Work: The authors plan to extend T-Recs into a self-adapting system that can automatically tune its parameters for a given set of training documents. They also suggest that their framework can be used to evaluate a variety of approaches, from top-down to bottom-up, and that developers can focus on specific metrics relevant to their application domain.

Summary updated

BibTeX
@inproceedings{shahab2010open,
  author = {Asif Shahab and Faisal Shafait and Thomas Kieninger and Andreas Dengel},
  title = {{An open approach towards the benchmarking of table structure recognition systems}},
  booktitle = {Document Analysis Systems},
  year = {2010},
  doi = {10.1145/1815330.1815345},
}
Figure