TFLOP: Table Structure Recognition Framework with Layout Pointer Mechanism

Minsoo Khang, Teakgyu HongIJCAI 2024

Abstract

Table Structure Recognition (TSR) is a critical task in document analysis, yet traditional "dual decoder" models often suffer from a "matching gap"—a misalignment between predicted logical structures and physical cell coordinates. We present TFLOP (TSR Framework with LayOut Pointer), which addresses this by replacing bounding box regression with a direct pointing mechanism. By linking HTML structure tags directly to pre-detected text regions, TFLOP eliminates the need for brittle post-processing. Our framework achieves state-of-the-art results on PubTabNet (96.66 TEDS) and FinTabNet (99.45 TEDS), demonstrating superior robustness in industrial and noisy scenarios.

The Matching Gap Problem

Converting table images into machine-readable formats like HTML typically involves two parallel tasks: logical structure prediction (HTML tags) and physical structure prediction (cell bounding boxes). The "Matching Gap" arises during the integration phase. To populate the HTML structure with content, systems must map predicted bounding boxes to text regions identified by OCR. Even minor inaccuracies in coordinate prediction can lead to text being assigned to the wrong cell or omitted entirely, necessitating complex and manual post-processing.

Comparison of the traditional dual decoder framework (a) and the TFLOP approach (b), highlighting how the layout pointer eliminates the 'matching gap' during post-processing.

TFLOP bypasses this issue by treating text regions as fixed anchors. Instead of hallucinating coordinates, the model "points" to existing candidate text regions to fill the generated HTML tags.

The TFLOP Architecture

The TFLOP framework consists of several integrated components designed to leverage both visual and layout information.

The TFLOP architecture, integrating an Image Encoder, Layout Encoder, and a Logical Structure Decoder with a Layout Pointer mechanism.

Hybrid Encoding and Logical Decoding

The system utilizes a Swin Transformer as an Image Encoder to extract global visual features. Simultaneously, a Layout Encoder embeds the coordinates of candidate text regions (provided by OCR or PDF parsers). A key feature is the use of ROIAlign, which crops visual features specifically from these text regions, ensuring the model's attention is grounded in the physical layout.

The Logical Structure Decoder, a Transformer-based module, generates a sequence of HTML structure tokens. For every data tag (e.g., <td>) produced, the Layout Pointer identifies the corresponding text region(s) from the input list, effectively building the table structure and content mapping in a single pass.

Span-aware Contrastive Supervision

To handle complex tables with merged cells, TFLOP employs a novel Span-aware Contrastive Supervision loss. Unlike standard contrastive learning, this approach uses a "span-coefficient" to weight text region overlaps. This helps the model learn the nuanced relationships between spanning cells and their constituent text blocks, improving accuracy in multi-row or multi-column layouts.

Experimental Results

TFLOP was evaluated on major benchmarks including PubTabNet, FinTabNet, and SynthTabNet.

  • State-of-the-Art Performance: TFLOP set new benchmarks with a 96.66 TEDS score on PubTabNet and 99.45 on FinTabNet, outperforming previous models like VAST and DRCC.
  • Industrial and Cross-lingual Robustness: On a self-annotated Korean table dataset, TFLOP achieved 92.0% QA accuracy, significantly surpassing GPT-4V (78.86%).
  • Noise Handling: The model demonstrated high robustness against document noise. By incorporating a binary classifier within the pointer mechanism, TFLOP successfully ignores "watermark" text regions that do not belong to the actual table structure.

Conclusion and Future Work

TFLOP demonstrates that "pointing" is a more robust strategy than coordinate prediction for table recognition. By grounding logical structure in existing layout anchors, it eliminates the most common source of error in TSR systems. While the framework is currently dependent on the quality of initial OCR/text detection, its success suggests broad applicability to other document AI tasks, such as form understanding and general layout analysis. Future research will focus on improving performance on extremely sparse tables and reducing the dependency on external text detection.

Summary updated

BibTeX
@inproceedings{khang2024tflop,
  author = {Minsoo Khang and Teakgyu Hong},
  title = {{TFLOP: Table Structure Recognition Framework with Layout Pointer Mechanism}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}

Loading related content...

Figure