Abstract
Table Structure Recognition (TSR) is a critical task in document analysis, requiring both the identification of logical cell relationships and the precise localization of physical cell boundaries. While recent image-to-text models excel at generating logical structures (e.g., HTML), they often struggle with physical coordinate precision. This paper introduces VAST (Visual-Alignment Sequential Coordinate Modeling), an end-to-end framework that treats cell coordinates as a language sequence and introduces a novel visual-alignment loss. By anchoring abstract logical tags to precise image pixels, VAST achieves state-of-the-art performance across multiple benchmarks, significantly improving the accuracy of both logical and physical table reconstruction.
Introduction
Converting table images into machine-readable formats like HTML or Excel remains a significant challenge in the field of computer vision. This process, known as Table Structure Recognition (TSR), involves two distinct sub-tasks: logical structure recognition (identifying rows, columns, and spanning cells) and physical structure recognition (determining the exact bounding boxes for every cell).
Existing image-to-text models have demonstrated strong performance in logical structure generation. However, these models frequently fail to provide the spatial precision required for physical recognition. Because they often rely on global representations to predict coordinates, they lack the local visual focus necessary for pixel-perfect bounding boxes. Even minor inaccuracies in these boxes can cause downstream Optical Character Recognition (OCR) systems to fail by truncating text or capturing neighboring cell data. VAST addresses these limitations by modeling coordinates sequentially and aligning them directly with visual features.
Methodology
Architecture Overview
The VAST framework is composed of three primary modules designed to bridge the gap between logical structure and physical layout:
- CNN Image Encoder: Utilizing a modified ResNet architecture, this component extracts high-level visual features from the input table image.
- HTML Sequence Decoder: A Transformer-based decoder that predicts the table's logical structure as a sequence of HTML tags (e.g.,
<tr>,<td>). - Coordinate Sequence Decoder (CSD): Unlike traditional regression-based approaches that predict bounding box coordinates simultaneously, the CSD treats coordinates ($x_{left}, y_{top}, x_{right}, y_{bottom}$) as a language sequence. By predicting these values one by one, the model learns the inherent geometric dependencies and constraints of a bounding box (e.g., $x_{right} > x_{left}$).
Visual-Alignment Loss
A core innovation of VAST is the introduction of a visual-alignment loss during training. Using a contrastive learning approach (InfoNCE), the model is trained to maximize the similarity between the internal representation of a cell (derived from the HTML decoder) and the actual visual features at that cell's location in the image (extracted via ROIAlign). This mechanism ensures that abstract logical tags are grounded in concrete image pixels, providing the "local" detail missing from previous global models.
Parallel Decoding Efficiency
While VAST treats coordinates as sequences, it avoids the latency associated with decoding boxes one by one in a single long string. Instead, the model collects all non-empty cell representations and decodes their respective coordinate sequences in parallel, maintaining high efficiency without sacrificing the benefits of sequential modeling.
Experimental Results
Datasets and Metrics
VAST was evaluated on a comprehensive suite of TSR benchmarks, including PubTabNet, FinTabNet, ICDAR2013, IC19B2M, SciTSR, and PubTables-1M. Performance was measured using standard metrics such as Tree Edit-Distance-based Similarity (TEDS) and Average Precision (AP).
Performance Highlights
VAST established new state-of-the-art (SOTA) results across nearly all tested benchmarks:
- Physical Precision: On the PubTabNet dataset, cell detection accuracy ($AP_{50}$) improved from the previous baseline of 82.1% (TableFormer) to 94.8%.
- Logical Accuracy: The model achieved a TEDS score of 98.21 on FinTabNet, outperforming all existing methods.
- Robustness: Qualitative analysis indicates that VAST is significantly more effective at handling "misaligned" tables where text is not perfectly aligned to a grid—a common failure point for traditional "split-and-merge" models.
Discussion and Limitations
VAST demonstrates that treating coordinates as a "grammar" and enforcing visual alignment significantly enhances TSR performance. However, the authors identify two primary areas for improvement:
- Inference Speed: As an auto-regressive model, VAST is slower than non-sequential methods, operating at approximately 1.38 FPS. This may present a bottleneck for real-time, large-scale document processing.
- Sequence Redundancy: The use of standard HTML tags results in long, repetitive sequences (e.g., frequent
<td></td>tags), which increases computational overhead and memory usage.
Conclusion
VAST represents a significant advancement in Table Structure Recognition by successfully integrating sequential coordinate modeling with visual-alignment constraints. By bridging the gap between global logical structure and local physical detail, it provides a robust solution for complex document reconstruction. Future research will focus on optimizing sequence lengths and exploring non-autoregressive decoding strategies to improve inference speed while maintaining high precision.
Summary updated