TL;DR
Vision-Language Models (VLMs) are getting good at understanding images and text, but they struggle with multi-tabular data presented visually, like on web pages. This paper introduces MTabVQA, a new benchmark and dataset to specifically test and improve VLMs' ability to reason across multiple tables in images, a crucial skill for real-world applications.
What Problem Does This Paper Address?
Modern AI models, particularly Vision-Language Models (VLMs), are increasingly capable of interpreting visual information and text. However, a significant challenge remains in their ability to robustly interpret and reason over multi-tabular data presented as images. This is a common scenario in real-world applications, such as analyzing financial reports, e-commerce websites, or digital documents, where crucial information is often spread across several tables with varying layouts. Existing benchmarks typically focus on single tables or non-visual data (text/structured formats), leaving a critical gap in evaluating a VLM's capacity to:
- Understand diverse visual table layouts presented as images.
- Parse and correlate information across multiple, physically separate tables.
- Execute multi-hop reasoning that requires integrating information from these visual tables.
This limitation hinders the development of VLMs that can effectively act as intelligent agents for tasks requiring complex data interpretation from visual sources.
Methods
The core contribution of this paper is the MTabVQA dataset and benchmark. The construction of MTabVQA involved a multi-stage pipeline:
- Data Sourcing and Sampling: The authors identified and filtered data from existing text-to-SQL and multi-table QA datasets (Spider, QFMTS, BIRD, MultiTabQA, ATIS, MiMoTable). They specifically selected subsets that involved multi-table joins or inherently required multi-table reasoning. Relational sampling was employed to create smaller, interconnected table samples.
- Visual QA Generation:
- SQL-to-Question: Complex SQL queries from the source datasets were converted into natural language questions. Ground truth answers were generated by executing these queries on the sampled tables.
- Taxonomy-Guided Generation: An LLM was used to generate novel QA pairs based on a predefined taxonomy of reasoning patterns (e.g., fact-checking, aggregation, ranking).
- Visual Rendering: The sampled tabular data for each QA pair was rendered into diverse images, incorporating variations in structure, layout, color schemes, typography, and styling elements to mimic real-world table appearances.
- Verification and Filtering: An automated process involving three LLM agents, followed by human verification, was used to ensure the quality, multi-table necessity, and correctness of the generated QA pairs.
Additionally, the paper introduces MTabVQA-Instruct, a large-scale instruction-tuning dataset derived from the same sources, to facilitate fine-tuning VLMs. They also fine-tuned a VLM called TableVision on this dataset.
Datasets
- MTabVQA Benchmark: A novel benchmark comprising 3,745 question-answer pairs designed for evaluating multi-hop reasoning over visually rendered tables. It is split into four sub-datasets based on the original data sources: MTabVQA-Spider, MTabVQA-Query, MTabVQA-ATIS, and MTabVQA-MiMo.
- MTabVQA-Instruct: A large-scale instruction-tuning dataset with 15,853 examples, used to improve VLM performance on multi-tabular visual reasoning.
- Source Datasets: The construction utilized data from existing benchmarks including Spider, QFMTS, BIRD, MultiTabQA, ATIS, and MiMoTable.
Key Insights and Ideas
- The "Visual Multi-Tabular Reasoning" Gap: The paper convincingly argues that existing benchmarks are insufficient for evaluating VLMs on the critical task of reasoning across multiple tables presented as images. This is a significant hurdle for real-world applications.
- Visual Diversity is Crucial: Simply converting tables to text is not enough. The MTabVQA benchmark's emphasis on visually rendered tables with diverse styling forces models to grapple with layout and OCR challenges, mimicking real-world complexity.
- Instruction Tuning Matters: Fine-tuning open-source VLMs on the MTabVQA-Instruct dataset significantly boosts their performance on multi-tabular visual reasoning, even outperforming larger proprietary models in some cases. This highlights the power of targeted training data.
- Data Diversity Trumps Scale Alone: When fine-tuning, the diversity and alignment of the training data are more critical than sheer volume. A large but narrowly focused dataset (MultiTabQA subset) led to poorer generalization than smaller, more diverse subsets.
- Numerical Reasoning is a Weak Spot: Current VLMs, even powerful ones like GPT-4.1, struggle with tasks requiring explicit numerical reasoning (aggregation, counting) across tables, indicating a key area for future research.
Key Results
- Benchmarking VLMs: State-of-the-art VLMs (proprietary and open-source) showed significant performance limitations on the MTabVQA benchmark in a zero-shot setting. For example, LLaVA-One-Vision achieved only 2.2% EM and 16.7% F1 overall, while even GPT-4.1 reached 37.0% EM and 61.7% F1 overall, indicating room for improvement.
- TableVision Performance: The fine-tuned model, TableVision, achieved the highest overall performance on MTabVQA (43.4% EM, 68.2% F1), surpassing GPT-4.1 in some sub-datasets. This demonstrates the effectiveness of MTabVQA-Instruct for targeted fine-tuning.
- Post-Training Strategies: Supervised Fine-Tuning (SFT) with LoRA yielded the most significant improvements for VLM performance on multi-tabular reasoning, outperforming Chain-of-Thought (CoT) prompting and Group Relative Policy Optimization (GRPO).
- Reasoning Category Performance: VLMs excel at descriptive analysis and fact-checking tasks but show weaker performance on numerical reasoning tasks like aggregation and counting. GPT-4.1 and Gemini-2.0-Flash had F1 scores below 0.57 for these numerical tasks.
Limitations and Future Work
- English-Only: The current MTabVQA benchmark is primarily English-centric, limiting its applicability for multilingual contexts. The authors suggest extending it to include multilingual tables and queries.
- Synthetic Table Layouts: While the benchmark incorporates visual diversity, it could be expanded to include more complex real-world table layouts, such as those with extensive cell merging, embedded charts, or noise from scanned documents.
- Limited Annotation: Human verification was performed by a single annotator, which could introduce minor errors.
- Future Work: The authors plan to explore more programmatically generated or real-world sourced table images with greater visual diversity and degradation to rigorously test VLM capabilities. They also aim to improve quantitative reasoning abilities in complex, multi-table visual contexts.
Summary updated