Introduction
Researchers from IIT Jodhpur have introduced TabComp, a new dataset designed to push the boundaries of how AI "reads" tables. Unlike traditional methods that rely on error-prone Optical Character Recognition (OCR), TabComp focuses on OCR-free, generative table reading comprehension, challenging models to understand complex layouts and handwritten notes to provide natural language answers.
The Challenge of Visual Table Reading
While Large Language Models (LLMs) excel at processing text, they often struggle when text is embedded within a table image. Conventional systems typically employ a two-step process: an OCR engine converts the image to text, which a model then processes. However, OCR frequently fails when encountering complex layouts, merged cells, or handwritten annotations common in industrial documents. If the OCR step fails, the downstream model cannot provide an accurate answer. TabComp addresses this by fostering OCR-free models that perceive visual structure and text simultaneously, providing a benchmark for human-like, context-aware answers.
Methodology
The development of TabComp involved a multi-stage pipeline to transform raw document images into a high-quality Visual Question Answering (VQA) dataset.
- Extraction: Table and list images were manually curated from the DocVQA dataset, focusing on "messy" real-world industrial reports.
- Generative Refinement: Ground truth answers were manually re-annotated to be descriptive and generative (e.g., "The total amount listed in the final column is 42 dollars") rather than simple one-word responses.
- Multi-Format Support: The dataset is released in two formats: a Standard QA format for models like Donut and an Instruction-Tuning format for universal models like UReader.
The TabComp Dataset
The primary contribution, the TabComp dataset, consists of 3,318 images and 19,610 questions. It is uniquely characterized by its inclusion of industrial documents containing a mix of printed and handwritten text—a significant challenge for existing models.
Compared to datasets like VisualMRC, TabComp features longer, more complex generative answers, averaging 11.48 tokens per answer versus 9.53 in VisualMRC.
Key Insights and Results
The study yielded several critical observations regarding visual reasoning:
- End-to-End Advantage: Models trained to extract data and generate answers simultaneously outperformed those fine-tuned on specific sub-tasks, suggesting that understanding table context aids data localization.
- Catastrophic Forgetting: Fine-tuning specialized models often led to a loss of general language skills, resulting in repetitive or nonsensical answers.
- Visual Nuance: OCR-free models better preserve document nuance by utilizing visual cues like cell proximity and line formatting that OCR typically discards.
Benchmarking results showed that Donut-base (fine-tuned) was the top performer with a BLEU-4 score of 42.69 and a BERTScore of 83.38. While the modern UReader model struggled more with specific tabular structures (BLEU-4 of 28.14), its high CIDEr score (210.77) indicated it could still capture the "essence" of the required information.
Limitations and Future Directions
A notable limitation of TabComp is the lack of text localization (bounding box) information, making it difficult to diagnose whether failures stem from visual perception or logical reasoning. Future research will focus on integrating spatial awareness into OCR-free models, mitigating catastrophic forgetting, and expanding the dataset to include more diverse document types like financial ledgers and scientific charts.
Summary updated