TL;DR
Existing tabular question-answering benchmarks predominantly evaluate models on single, small tables, obscuring how poorly large language models (LLMs) handle real-world relational databases. To address this, the authors introduce TQA-Bench, a multi-table QA benchmark featuring scalable context lengths (8K to 64K tokens) constructed via foreign-key-preserving sampling and template-driven symbolic extensions. Systematic evaluation of 28 open- and closed-source LLMs reveals severe drops in accuracy as schemas expand and context lengths grow, particularly for complex statistical queries.
What Problem Does This Paper Address?
Most enterprise data lives in normalized, multi-table relational databases linked by foreign keys (e.g., in finance, healthcare, and logistics). However, current Table QA benchmarks suffer from three major shortcomings:
- Single-table bias: Datasets like WikiTableQuestions, TabFact, and FinQA overwhelmingly evaluate single, flat tables, completely bypassing cross-table joins.
- Tiny context footprints: Existing benchmarks typically feature tables with only dozens of rows (~500–1,500 tokens), failing to test modern LLMs' long-context capabilities.
- Data leakage and pattern matching: Relying heavily on Wikipedia risks contamination in pre-training data, allowing models to exploit statistical shortcuts rather than performing genuine multi-hop relational reasoning.
Methods
The authors build TQA-Bench through a principled, four-stage pipeline:
- Foreign-Key-Preserving Sampling: To fit massive databases into controllable context windows (8K, 16K, 32K, and 64K tokens), the authors topologically sort database tables along directed acyclic foreign-key graphs. Rows are sampled parent-first, restricting child tables to valid foreign keys to maintain referential integrity. Binary search on the sample rate tunes instances to target token budgets.
- Symbolic Question Generation: Following the GSM-Symbolic paradigm, questions are generated from 140 parameterized templates (2 per subcategory across 10 databases) and instantiated using Python scripts to compute exact ground truths.
- Task Categorization: Tasks span three levels across seven formal relational-algebra subcategories:
- Lookup: Entity Lookup (EL), Top Selection (TS)
- Aggregation: Count (CNT), Sum (SUM), Average (AVG)
- Complex Calculation: Composite Comparison (CC), Correlation (COR)
- Controlled Multiple-Choice Evaluation: Questions are framed as multiple-choice questions (MCQs) with rule-based distractors (e.g., ground-truth multiples of $0.25\times, 2.0\times, 3.0\times$), ensuring reproducible, deterministic exact-match evaluation.
+------------------------------------+
| Topological Sort (DAG) of Tables |
+-----------------+------------------+
|
v
+-----------------+------------------+
| Foreign-Key Consistent Row Sampling| --> Binary search for 8K-64K tokens
+-----------------+------------------+
|
v
+-----------------+------------------+
| Parameterized Symbolic Templates | --> Python-computed ground truths & MCQs
+------------------------------------+
Datasets
The authors deliberately exclude Wikipedia-based tables, selecting 10 real-world relational databases across three non-Wikipedia sources:
- BIRD: 7 curated databases with clean foreign keys (airline, food_inspection, movie, music_tracker, restaurant, university, cookbook).
- Data.gov: 2 public-sector datasets (Water Quality Data, Food Facility Inspections).
- World Bank: 1 global indicator dataset (Global Biodiversity).
Together, these databases yield 400 sampled database instances and 56,000 question instances across 8K, 16K, 32K, and 64K token context budgets.
Key Insights and Ideas
- Single- vs. Multi-Table Performance Gap: LLMs score up to 20% higher when tables are pre-joined into a single denormalized table compared to processing normalized multi-table inputs. Merging tables helps, but requires careful column renaming to preserve semantic clarity.
- Serialization Matters: Markdown consistently beats CSV, JSON, and HTML in direct LLM prompting. JSON is structurally the weakest, and HTML balloons context size by roughly $3\times$ compared to CSV.
- The Pitfall of Distillation: Distilled reasoning models (e.g., DeepSeek-R1-Distill-Qwen-7B/14B) severely underperform their non-distilled base counterparts on long tabular contexts, frequently falling into repetitive generation loops.
- Direct Prompting vs. Text2SQL: LLM-based Text2SQL pipelines are far more stable across context lengths because they only see schema definitions. However, Text2SQL struggles drastically on complex queries (e.g., scoring near 0% on Pearson correlations because models cannot easily compose correct multi-line statistical SQL or output raw mathematical formulas instead of SQL).
- Specialized Models Disappoint: Table-specific models like TableLlama and TableGPT2 fail to match general frontier instruct models, suffering from rigid output formatting and limited general reasoning flexibility.
Key Results
- Overall Rankings: The full DeepSeek-R1 (671B MoE) achieved top overall accuracy (89.39% macro average), followed closely by GPT-o3-mini (84.91%) and GPT-4o (70.81%). Chat-oriented models generally failed the instruction format, averaging $<25\%$.
- Context Length Degradation: On Qwen2.5-7B-Instruct, increasing context length from 8K to 64K dropped overall accuracy from 49.86% down to 33.43%.
- Task-Specific Fragility: Lookups remain relatively robust across scales, but aggregations and correlations degrade quickly. On 64K correlation (COR) tasks, even the frontier GPT-5.1 achieved only 53.54% accuracy with direct prompting (though incorporating a Python code interpreter boosted this to 60.53%).
Limitations and Future Work
- Strict Multiple-Choice Format: While manual inspection of 50 errors showed models genuinely failed the reasoning rather than the formatting, the MCQ design does not award partial credit for intermediate arithmetic steps.
- Fully Relational Only: TQA-Bench is limited to structured relational tables; it does not evaluate semi-structured web tables or document stores (e.g., nested JSON).
- Text-Only Modality: The benchmark only covers text serializations and omits visual multi-table question answering (e.g., multimodal sheet comprehension).
Summary updated