Introduction
Table Representation Learning (TRL) models—like TAPAS, TAPEX, or even ChatGPT—are typically pre-trained on massive open-domain datasets. While they perform impressively on standard benchmarks like Spider, their performance often fails to translate to proprietary corporate data, which may have different schemas, data distributions, or quality issues.
For engineers, selecting the right model is a headache. Manually crafting thousands of "Question-Answer" pairs to test a model on a private database is prohibitively expensive and slow. Furthermore, existing evaluation metrics are often too "all-or-nothing," failing to distinguish between a model that is slightly off and one that is completely hallucinating.
To solve this, researchers introduced QATCH (Query-Aided TRL Checklist), a new toolbox that automatically generates custom benchmarks to test how well AI models understand your specific tabular data.
The QATCH Pipeline
QATCH moves beyond static, open-source benchmarks by evaluating TRL models on proprietary datasets using granular metrics. The framework operates in a three-step automated pipeline:
- QATCH-Generate: Instead of relying on human-written questions, QATCH uses a SQL query engine to automatically generate a "checklist" of tests based on the input table's schema and content. It creates SQL queries of varying complexity (e.g., simple projections, filters, or sorting) and uses templates to convert these into Natural Language (NL) questions.
- TRL Model Prediction: These NL questions are fed into TRL models for two tasks: Question Answering (QA), where the model returns data cells, and Semantic Parsing (SP), where the model returns a SQL query.
- QATCH-Evaluate: Rather than simple accuracy, QATCH uses five "cross-task" metrics to reveal exactly where a model’s logic fails.
Evaluation Metrics
The evaluation phase is the "secret sauce" of QATCH, breaking down performance into five distinct categories:
- Cell Precision & Recall: Did the model find the right values, and did it find all of them?
- Tuple Constraint: Did the model preserve the relational structure (e.g., keeping the right columns with the right rows)?
- Tuple Cardinality: Did it return the correct number of results?
- Tuple Order: For "Order By" questions, is the ranking correct?
Experimental Setup and Key Insights
The researchers evaluated their tool using 8 proprietary tables sourced from Kaggle, representing diverse domains (E-commerce, Finance, Medicine) and varying in size from 303 to 1 million rows. They compared these against the standard Spider benchmark to highlight the performance gap when moving to custom data.
Key Findings: * The Benchmark Illusion: Models that perform exceptionally well on the Spider benchmark often see a significant performance drop when faced with proprietary data. Static benchmarks aren't enough for real-world deployment. * Precision vs. Recall Trade-offs: Traditional TRL models like TAPAS and TAPEX tend to be conservative. In the QA task on proprietary data, TAPAS-Large-WTQ showed strong Cell Precision (up to 0.72) but dismal Cell Recall (around 0.12), meaning it only finds a tiny fraction of the correct answers. * The "Relational" Struggle: Most models struggle with Tuple Constraints. They might find the right individual cells but fail to return them in a coherent table structure that matches the ground truth. * ChatGPT's Performance: In the Semantic Parsing (Text-to-SQL) task, ChatGPT 3.5 was the top performer, achieving an average Tuple Constraint score of 0.90+ across several categories. However, it suffers from hallucination in its decoder architecture, leading to lower precision in some cases compared to specialized models, and it specifically struggled with tables dominated by numerical data.
Efficiency and Future Work
QATCH can generate a comprehensive test suite for a new dataset in just a few seconds, making continuous evaluation feasible for production databases.
The authors acknowledge a few current limitations:
- Single-Table Focus: QATCH currently focuses on single-table queries and does not yet support complex SQL
JOINoperations across multiple tables. - Token Limits: TRL models have strict input limits (e.g., 512 or 1024 tokens). To fit large proprietary tables into these models, QATCH has to sample rows and columns, which might miss some edge cases.
Future iterations of QATCH aim to support more models (like LLaMA 2), more complex SQL features (like GROUP BY and nested queries), and multi-table relational constraints.
Summary updated