TL;DR
This paper presents a system for answering questions over tabular data using the TableGPT2 model, achieving strong performance on the SemEval-2025 Task 8. The approach leverages a specialized tabular encoder and a prompt generation strategy to effectively translate natural language questions into executable Python code for data extraction.
What Problem Does This Paper Address?
The paper tackles the challenge of Question Answering (QA) over tabular data, a critical task for extracting insights from structured information. While LLMs have revolutionized NLP, effectively applying them to structured data like tables remains an open problem. Traditional methods often rely on complex semantic parsing to SQL or SPARQL, which struggle with natural language nuances and multi-step reasoning. The authors highlight the need for robust benchmarks and models that can bridge the gap between natural language queries and structured data retrieval, especially for complex, multi-step reasoning tasks.
Methods
The core of the system is the TableGPT2-7B model, a language model specifically designed for tabular data. Its key innovation lies in a tabular encoder that processes entire tables, generating compact embeddings for each column. This encoder is combined with a modality module that integrates these tabular embeddings with textual input embeddings, allowing the model to understand table structure and semantics.
The system also employs a Prompt Generator that crafts prompts for the LLM. These prompts include a general task description, examples of question-table-answer triplets, and the specific question and table summary for the LLM to process.
Finally, an Automatic Code Execution module is used. The LLM generates Python code (using the Pandas library) to answer the question. This code is then executed, and if errors occur, the system provides feedback to the LLM for correction, enhancing robustness.
Datasets
The primary dataset used for evaluation is DataBench, which is part of SemEval-2025 Task 8. DataBench consists of numerous publicly available datasets, categorized by domain (Business, Health, Social, Sports, Travel). Each dataset includes natural language questions, relevant column information, and corresponding answers. The paper also mentions a "DataBench Lite" benchmark, suggesting a variation or subset of the main DataBench.
Key Insights and Ideas
- Specialized Tabular Understanding is Crucial: The success of TableGPT2 highlights that models need specific architectural components (like the tabular encoder) to effectively process and understand tabular data, going beyond generic text processing.
- Prompt Engineering Matters: The prompt generator plays a significant role in guiding the LLM to produce accurate and executable code, demonstrating the importance of well-crafted prompts for task-specific LLM applications.
- Code Execution for Robustness: Automatically executing the generated code and providing error feedback to the LLM is a practical strategy to improve accuracy and handle potential LLM generation issues.
- Parameter Scale vs. Training Strategy: While larger models generally perform better, the paper suggests that factors like data quality and training strategy (as seen with Deepseek-Coder) are equally important for achieving optimal performance.
Key Results
The YNU-HPCC system, powered by TableGPT2-7B, achieved competitive results on SemEval-2025 Task 8.
- DataBench Accuracy: 65.64%
- DataBench Lite Accuracy: 66.22% Both scores exceeded the baseline. The paper also notes that TableGPT2-7B achieved the highest performance scores across all subtasks in the competition. When comparing against other models like Deepseek-Coder-6.7B-Base (0.51% accuracy), TableGPT2-7B showed a notable performance advantage.
Limitations and Future Work
The authors acknowledge that while increasing parameter scale generally improves performance, this improvement is not endless. They suggest that future work could involve adjusting basic parameters or exploring models with even larger parameter counts to further optimize experimental results. The paper also implies that the effectiveness of the current approach could be further enhanced by optimizing other factors beyond just model size, such as data quality and training strategies.
Summary updated