Abstract
While Text-to-SQL systems excel at generating structured queries, the subsequent task of transforming raw tabular results into natural language responses (NLRs) remains a significant challenge. This "last mile" problem is often characterized by hallucinations, information loss, and poor formatting. This paper introduces NLR-BIRD, a dedicated benchmark for data narration, and Combo-Eval, a novel hybrid evaluation framework. By leveraging extremity thresholds from traditional metrics to filter clear-cut cases, Combo-Eval reduces LLM evaluation costs by up to 61% while maintaining high alignment with human judgment. Our findings highlight the limitations of current frontier models in handling large-scale tabular data and the importance of reasoning capabilities in reference-less evaluation.
Introduction
In modern AI assistants, Text-to-SQL technology serves as a vital bridge between natural language queries and relational databases. However, research has predominantly focused on the accuracy of SQL generation, overlooking the critical step of narrating the resulting data for the end-user. Large Language Models (LLMs) frequently struggle with this task, particularly as table size increases. Current evaluation methods present a trade-off: traditional metrics like ROUGE are often too simplistic, while high-end LLM judges are computationally expensive and slow. This study addresses these gaps by providing a robust dataset and a cost-effective evaluation framework.
The NLR-BIRD Dataset
To facilitate research in data narration, we present NLR-BIRD, a benchmark derived from the BIRD (Big Bench for Large-scale Database Grounding) dataset. NLR-BIRD comprises 1,468 samples across 11 diverse domains, including finance, sports, and education. Unlike existing Table QA datasets that focus on specific cell extraction, NLR-BIRD emphasizes the holistic narration of entire result sets. The dataset includes human-labeled ground truth narrations for tables with up to 500 cells, providing a high-quality reference for model evaluation.
The Combo-Eval Framework
We propose Combo-Eval, a three-tiered hybrid evaluation approach designed to balance accuracy and efficiency. The framework operates by first computing traditional NLP metrics (e.g., ROUGE-1 recall) to establish "extremity thresholds." Responses that fall into the highest or lowest scoring brackets are automatically classified as correct or incorrect, respectively. Only the ambiguous "middle-ground" samples are passed to a high-capacity LLM judge for final assessment.
This method was tested in two scenarios: GT (Ground Truth available) and UQDB (User Question and Database result set only), the latter simulating production environments where reference answers are unavailable.
Results and Performance Analysis
Our evaluation reveals that Combo-Eval significantly enhances efficiency, reducing expensive LLM judge calls by 61.43% in the GT scenario and 24.48% in the UQDB scenario. Crucially, this efficiency does not come at the cost of accuracy; Combo-Eval demonstrated higher alignment with human judgment than standalone metrics or LLM judges, especially when paired with smaller, more efficient models.
Performance analysis shows that even frontier models like Llama-3.1-405B and GPT-4o experience significant degradation as table size increases. For result sets exceeding 50 cells, GPT-4o’s acceptable response rate dropped from 91% to 30%. The primary failure mode was identified as incomplete information, where models omitted rows or columns from larger tables.
Conclusion and Future Work
This work underscores the complexity of the "last mile" in Text-to-SQL systems. While Combo-Eval provides a scalable solution for evaluation, significant challenges remain in model generation for large-scale data. Future research will focus on advanced summarization strategies for massive tables, the integration of arithmetic reasoning for complex data sets, and fine-tuning models specifically on the NLR-BIRD dataset to improve their narrative capabilities.