Treading Towards Privacy-Preserving Table Structure Recognition

Sachin Raja, Ajoy Mondal, C.v. JawaharWACV 2025

TL;DR

This paper introduces TabGuard, a novel end-to-end framework for privacy-preserving Table Structure Recognition (TSR). By masking table content locally before sending it to a server for processing, TabGuard enables accurate table structure extraction from sensitive documents without compromising data privacy, achieving state-of-the-art performance.

What Problem Does This Paper Address?

Table Structure Recognition (TSR) is crucial for extracting structured data from documents, but it often involves processing sensitive information found in financial or healthcare records. Existing TSR methods, typically relying on deep learning models, require sending document images to a server, posing significant privacy risks. While some privacy-preserving techniques exist, they often fall short in addressing the specific challenges of TSR, such as handling dense tables with complex layouts and extreme aspect ratios, or they introduce substantial computational overhead. This paper aims to bridge this gap by developing a privacy-preserving TSR system that is both effective and efficient.

Methods

TabGuard employs a multi-stage approach:

  • Client-Side Content Masking: A lightweight, language-agnostic algorithm masks the table's content by replacing it with blacked-out boxes. This process uses standard image processing techniques and is designed to be efficient for on-device execution.
  • Server-Side Table Grid Approximator (TGA): This unsupervised module takes the masked table image and approximates the table's structural grid. It leverages text contour distribution and inter-contour spacing to identify potential row and column separators. This approximated grid provides crucial inductive bias for subsequent steps.
  • Server-Side Table Cell Crypt Network (TCCN): Built upon a ResNeXt-101 backbone, TCCN is a simplified Faster R-CNN that eliminates the need for a Region Proposal Network (RPN). Instead, it dynamically generates table-specific anchors using the TGA's approximated grid. This approach helps overcome challenges with dense tables and extreme aspect ratios. TCCN is trained with standard classification and regression losses, augmented by alignment and continuity losses for improved spatial accuracy.
  • Post-processing: A simple, dataset-agnostic post-processing step refines cell boundaries and assigns row/column spanning indices based on cell overlaps.

Datasets

The paper uses the following datasets for training and evaluation:

  • FinTabNet [71] (FTN): Used for training and evaluation.
  • SciTSR [7]: Used for training and evaluation.
  • PubTabNet [72] (PTN): Used for evaluation.
  • ICDAR-2013 [14] (IC-13): Used for evaluation.
  • ICDAR-2019 [11] (cTDaR): Used for evaluation.

The authors note that they do not compare on the WTW dataset because their method generates rectangular bounding boxes, which might not be suitable for misaligned or curved tables.

Key Insights and Ideas

  • Inductive Bias from Approximated Grid: The core innovation is using an approximated table grid (from TGA) to guide anchor generation. This provides strong inductive bias, significantly improving the accuracy of cell detection, especially in challenging scenarios.
  • Privacy Through Masking: Masking content locally on the client before sending to the server is a practical and effective way to achieve privacy in TSR, enabling the use of sensitive data without direct exposure.
  • OCR-Free Masking: The content masking algorithm is OCR-free, making it language-agnostic and robust to tables with or without explicit separators.
  • Dynamic Anchor Generation: Moving away from fixed anchor hyperparameters and generating dynamic, table-specific anchors based on the approximated grid leads to better coverage and alignment.
  • State-of-the-Art Performance with Privacy: TabGuard demonstrates that achieving high accuracy in TSR is possible even with privacy-preserving mechanisms, outperforming existing methods on several benchmarks.

Key Results

  • TabGuard achieves state-of-the-art performance on several benchmark datasets, including ICDAR-2013, cTDaR, and PubTabNet (S-TEDS and TEDS scores).
  • It significantly outperforms prior approaches on ICDAR-2013 and cTDaR, achieving margins of 0.3% and 1.3% in Cell Adjacency Relation F1 scores, respectively.
  • On SciTSR, TabGuard achieves an F1 score within 0.5% of the state-of-the-art while maintaining content privacy.
  • The TGA and anchor generation process reduce the number of anchors, leading to a 2.5x improvement in optimization performance.
  • The method is robust to variations in table density and complexity.

Limitations and Future Work

  • Scanned Images Focus: TabGuard is primarily designed for scanned images of cropped tables in a controlled 2D environment. It does not extend to tables captured by cameras, which may involve curvature, perspective distortions, or 3D effects.
  • Complex Embedded Entities: The approach does not support tables with complex embedded entities like nested tables, graphs, or images, which would require more advanced parsing techniques.
  • Skewed Images: While skew correction as a preprocessing step helps with minor misalignments, it may not be sufficient for severely distorted or significantly perspective-shifted images.

The authors suggest that future work could explore extending the framework to handle camera-captured tables and tables with complex embedded entities.

Summary updated

BibTeX
@inproceedings{raja2025treading,
  author = {Sachin Raja and Ajoy Mondal and C.V. Jawahar},
  title = {{Treading Towards Privacy-Preserving Table Structure Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure