SemiTabDETR: End-to-End Semi-Supervised Table Detection with Transformer-based Enhanced Query Approach

Tahira Shhezadi, Didier Stricker, Muhammad Zeshan AfzalResearch Square 2024 · DOI

Abstract

Table detection recognizes and accurately determines the position of tables within document images. This task involves both classification and precise localization of these table elements. Conventional table detection methods usually depend on extensive labeled data, creating a challenge in generating high-quality labels for training. To address this, many semi-supervised approaches are proposed. These methods utilize either CNN-based networks, which rely on anchor generation and NMS (Non-Maximum Suppression), or employ transformer-based models with performance linked to the quality of object queries. In this paper, we propose a transformer-based semi-supervised approach that improves the quality of object queries. We take high-level query features from unlabeled images and find similarities with decoder original queries by an enhanced query selection network. It provides high-quality, refined queries, allowing the model to make precise predictions or classifications with minimal labeled data. Results on benchmarks such as Publaynet, DocBank, PubTables, and ICDAR 19 have demonstrated that this innovative approach significantly outperforms traditional supervised and semi-supervised methods. On just 1% label data, our approach obtains 98.1% mAP, 96.4% mAP, and 82.8% mAP on PubTables, PubLayNet, and DocBank datasets, respectively. The state-of-the-art results show the effectiveness of our approach.

Introduction

Extracting data from tables is a cornerstone of automated document analysis, yet teaching a computer to accurately "see" and localize a table remains a significant challenge. Traditional deep learning models for table detection typically require massive amounts of manually labeled data—thousands of document images where humans have painstakingly drawn bounding boxes. This labeling process is slow, expensive, and prone to human error.

To mitigate the dependency on large-scale labeling, researchers have turned to semi-supervised learning, which leverages a small amount of labeled data alongside a large pool of unlabeled data. However, existing semi-supervised methods often rely on complex post-processing steps like Non-Maximum Suppression (NMS) to resolve duplicate predictions. While newer Transformer-based models (such as DETR) avoid NMS, they often struggle when labels are scarce because their internal "object queries"—the mathematical probes used to identify objects—remain low-quality. SemiTabDETR addresses these issues by refining object queries through a novel selection network and a hybrid matching strategy, achieving state-of-the-art performance with as little as 1% of the training labels.

Methodology

SemiTabDETR utilizes a "Teacher-Student" architecture designed for end-to-end semi-supervised learning. In this framework, a "Teacher" model generates "pseudo-labels" (predicted bounding boxes) on unlabeled images, which are then used to supervise the training of a "Student" model.

The architecture of SemiTabDETR, featuring the Teacher-Student framework and Enhanced Query Selection (EQS) module.

The framework incorporates several core innovations:

  1. Enhanced Query Selection (EQS): In standard Transformer detectors, object queries are often initialized as learned positional embeddings. SemiTabDETR’s EQS module extracts high-level features from unlabeled images and aligns them with the decoder's queries. By finding similarities between these features and the original queries, the EQS module provides refined, semantically richer queries that are more effective at identifying table structures even with minimal supervision.
  2. Hybrid Matching Strategy: To optimize training, the model employs two distinct matching mechanisms:
  3. One-to-Many Matching: This matches a single table to multiple queries, providing a stronger training signal and facilitating the generation of higher-quality pseudo-labels.
  4. One-to-One Matching: This ensures the final model produces exactly one box per table, maintaining the end-to-end nature of the Transformer and eliminating the need for NMS post-processing.
  5. Backbone and Transformer: The system utilizes a standard ResNet-50 backbone paired with a Deformable Transformer, enabling efficient handling of tables across various sizes and scales.

Experimental Setup

The researchers evaluated SemiTabDETR on four major document analysis benchmarks to test its robustness and accuracy:

  • PubTables-1M: A massive dataset designed for comprehensive table extraction tasks.
  • PubLayNet: A large-scale dataset focused on document layout analysis.
  • DocBank: A dataset featuring fine-grained labels for a wide variety of document elements.
  • ICDAR-19 (Track A): A classic competition benchmark for table detection in modern document formats.

Key Insights and Results

The primary insight of this research is that query quality is the bottleneck in semi-supervised Transformer-based detection. By using unlabeled data to refine these queries via the EQS module, SemiTabDETR bridges the performance gap between supervised and semi-supervised learning.

The quantitative results are remarkable:

  • The "1% Miracle": On the PubTables dataset, SemiTabDETR achieved 98.1% mAP (mean Average Precision) using only 1% labeled data. Remarkably, this outperformed a standard DETR model trained on 100% of the labels (96.6% mAP).
  • Consistent Performance: The model reached 96.4% mAP on PubLayNet and 82.8% mAP on DocBank with only 1% labels, significantly outperforming previous CNN-based semi-supervised methods.
  • Efficiency: On the ICDAR-19 benchmark, the model outperformed several established supervised and semi-supervised baselines in both precision and recall when using 50% of the labels.

By splitting the training into a pseudo-label generation stage (one-to-many) and a redundant prediction removal stage (one-to-one), the model achieves a rich training signal while maintaining a clean, NMS-free output.

Limitations and Future Work

While SemiTabDETR excels at detecting the outer boundaries of tables, it does not currently address Table Structure Recognition—the identification of internal rows, columns, and cells. Detecting the table boundary is a critical first step, but reconstructing the internal data remains a separate challenge.

Furthermore, the model's performance is sensitive to the "pseudo-labeling threshold" (set at 0.4 in this study). A threshold that is too high can slow down learning, while one that is too low may introduce "noisy" or incorrect boxes into the training process. Future research could explore dynamic or self-adjusting thresholds that adapt based on the model's confidence levels during different training stages.

Summary updated

Abstract

<title>Abstract</title> Table detection recognizes and accurately determines the position of tables within document images. This task involves both classification and precise localization of these table elements. Conventional table detection methods usually depend on extensive labeled data, creating a challenge in generating high-quality labels for training. To address this, many semi-supervised approaches are proposed. These methods utilize either CNN-based networks, which rely on anchor generation and NMS (Non-Maximum Suppression), or employ transformer-based models with performance linked to the quality of object queries. In this paper, we propose a transformer-based semi-supervised approach that improves the quality of object queries. We take high-level query features from unlabeled images and find similarities with decoder original queries by an enhanced query selection network. It provides high-quality, refined queries, allowing the model to make precise predictions or classifications with minimal labeled data. Results on benchmarks such as Publaynet, DocBank, PubTables, and ICDAR 19 have demonstrated that this innovative approach significantly outperforms traditional supervised and semi-supervised methods. On just 1% label data, our approach obtains 98.1% mAP, 96.4% mAP, and 82.8% mAP on PubTables, PubLayNet, and DocBank datasets, respectively. The state-of-the-art results show the effectiveness of our approach.
BibTeX
@inproceedings{shhezadi2024semitabdetr,
  author = {Tahira Shhezadi and Didier Stricker and Muhammad Zeshan Afzal},
  title = {{SemiTabDETR: End-to-End Semi-Supervised Table Detection with Transformer-based Enhanced Query Approach}},
  booktitle = {Research Square},
  year = {2024},
  doi = {10.21203/rs.3.rs-5305546/v1},
}
Figure