Improving Table Structure Recognition with Visual-Alignment Sequential Coordinate Modeling

Yongshuai Huang, Ning Lu, Dapeng Chen, Yibo Li, Zecheng Xie, Shenggao Zhu, Liangcai Gao, Wei Peng

Abstract

Table Structure Recognition (TSR) is a critical task in document analysis, requiring both the identification of logical cell relationships and the precise localization of physical cell boundaries. While recent image-to-text models excel at generating logical structures (e.g., HTML), they often struggle with physical coordinate precision. This paper introduces VAST (Visual-Alignment Sequential Coordinate Modeling), an end-to-end framework that treats cell coordinates as a language sequence and introduces a novel visual-alignment loss. By anchoring abstract logical tags to precise image pixels, VAST achieves state-of-the-art performance across multiple benchmarks, significantly improving the accuracy of both logical and physical table reconstruction.

Introduction

Converting table images into machine-readable formats like HTML or Excel remains a significant challenge in the field of computer vision. This process, known as Table Structure Recognition (TSR), involves two distinct sub-tasks: logical structure recognition (identifying rows, columns, and spanning cells) and physical structure recognition (determining the exact bounding boxes for every cell).

Existing image-to-text models have demonstrated strong performance in logical structure generation. However, these models frequently fail to provide the spatial precision required for physical recognition. Because they often rely on global representations to predict coordinates, they lack the local visual focus necessary for pixel-perfect bounding boxes. Even minor inaccuracies in these boxes can cause downstream Optical Character Recognition (OCR) systems to fail by truncating text or capturing neighboring cell data. VAST addresses these limitations by modeling coordinates sequentially and aligning them directly with visual features.

Methodology

Architecture Overview

The VAST framework is composed of three primary modules designed to bridge the gap between logical structure and physical layout:

  1. CNN Image Encoder: Utilizing a modified ResNet architecture, this component extracts high-level visual features from the input table image.
  2. HTML Sequence Decoder: A Transformer-based decoder that predicts the table's logical structure as a sequence of HTML tags (e.g., <tr>, <td>).
  3. Coordinate Sequence Decoder (CSD): Unlike traditional regression-based approaches that predict bounding box coordinates simultaneously, the CSD treats coordinates ($x_{left}, y_{top}, x_{right}, y_{bottom}$) as a language sequence. By predicting these values one by one, the model learns the inherent geometric dependencies and constraints of a bounding box (e.g., $x_{right} > x_{left}$).

Visual-Alignment Loss

A core innovation of VAST is the introduction of a visual-alignment loss during training. Using a contrastive learning approach (InfoNCE), the model is trained to maximize the similarity between the internal representation of a cell (derived from the HTML decoder) and the actual visual features at that cell's location in the image (extracted via ROIAlign). This mechanism ensures that abstract logical tags are grounded in concrete image pixels, providing the "local" detail missing from previous global models.

Parallel Decoding Efficiency

While VAST treats coordinates as sequences, it avoids the latency associated with decoding boxes one by one in a single long string. Instead, the model collects all non-empty cell representations and decodes their respective coordinate sequences in parallel, maintaining high efficiency without sacrificing the benefits of sequential modeling.

Experimental Results

Datasets and Metrics

VAST was evaluated on a comprehensive suite of TSR benchmarks, including PubTabNet, FinTabNet, ICDAR2013, IC19B2M, SciTSR, and PubTables-1M. Performance was measured using standard metrics such as Tree Edit-Distance-based Similarity (TEDS) and Average Precision (AP).

Performance Highlights

VAST established new state-of-the-art (SOTA) results across nearly all tested benchmarks:

  • Physical Precision: On the PubTabNet dataset, cell detection accuracy ($AP_{50}$) improved from the previous baseline of 82.1% (TableFormer) to 94.8%.
  • Logical Accuracy: The model achieved a TEDS score of 98.21 on FinTabNet, outperforming all existing methods.
  • Robustness: Qualitative analysis indicates that VAST is significantly more effective at handling "misaligned" tables where text is not perfectly aligned to a grid—a common failure point for traditional "split-and-merge" models.

Discussion and Limitations

VAST demonstrates that treating coordinates as a "grammar" and enforcing visual alignment significantly enhances TSR performance. However, the authors identify two primary areas for improvement:

  1. Inference Speed: As an auto-regressive model, VAST is slower than non-sequential methods, operating at approximately 1.38 FPS. This may present a bottleneck for real-time, large-scale document processing.
  2. Sequence Redundancy: The use of standard HTML tags results in long, repetitive sequences (e.g., frequent <td></td> tags), which increases computational overhead and memory usage.

Conclusion

VAST represents a significant advancement in Table Structure Recognition by successfully integrating sequential coordinate modeling with visual-alignment constraints. By bridging the gap between global logical structure and local physical detail, it provides a robust solution for complex document reconstruction. Future research will focus on optimizing sequence lengths and exploring non-autoregressive decoding strategies to improve inference speed while maintaining high precision.

Summary updated

Abstract

Table structure recognition aims to extract the logical and physical structure of unstructured table images into a machine-readable format. The latest end-to-end image-to-text approaches simultaneously predict the two structures by two decoders, where the prediction of the physical structure (the bounding boxes of the cells) is based on the representation of the logical structure. However, the previous methods struggle with imprecise bounding boxes as the logical representation lacks local visual information. To address this issue, we propose an end-to-end sequential modeling framework for table structure recognition called VAST. It contains a novel coordinate sequence decoder triggered by the representation of the non-empty cell from the logical structure decoder. In the coordinate sequence decoder, we model the bounding box coordinates as a language sequence, where the left, top, right and bottom coordinates are decoded sequentially to leverage the inter-coordinate dependency. Furthermore, we propose an auxiliary visual-alignment loss to enforce the logical representation of the non-empty cells to contain more local visual details, which helps produce better cell bounding boxes. Extensive experiments demonstrate that our proposed method can achieve state-of-the-art results in both logical and physical structure recognition. The ablation study also validates that the proposed coordinate sequence decoder and the visual-alignment loss are the keys to the success of our method.
BibTeX
@inproceedings{huang2023improving,
  author = {Yongshuai Huang and Ning Lu and Dapeng Chen and Yibo Li and Zecheng Xie and Shenggao Zhu and Liangcai Gao and Wei Peng},
  title = {{Improving Table Structure Recognition with Visual-Alignment Sequential Coordinate Modeling}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure