Abstract: Jyotika Singh, Weiyi Sun, Amit Agarwal, Viji Krishnamurthy, Yassine Benajiba, Sujith Ravi, Dan Roth. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track.
2025.
Introduction
In modern Text-to-SQL systems, the final step—Natural Language Representation (NLR)—is critical for user experience. It involves "narrating" raw database results into conversational responses. However, evaluating these narrations is challenging: traditional NLP metrics (e.g., ROUGE) often fail to capture faithfulness, while high-end LLM judges are too expensive for production scale. This paper introduces Combo-Eval, a hybrid framework designed to balance accuracy and cost-efficiency.
The Combo-Eval Framework
To address the limitations of existing evaluation methods, the authors propose a tiered approach. Combo-Eval utilizes standard metrics to handle clear-cut cases and reserves expensive LLM calls for ambiguous "gray area" samples.
The process involves:
- Metrics-as-a-Judge: Calculating a base metric (specifically ROUGE-1 recall).
- Extremity Thresholding: Making immediate decisions for very high (likely correct) or very low (likely incorrect) scores.
- LLM-as-a-Judge: Routing only the uncertain middle-ground samples to a judge LLM for a final verdict.
The framework supports two scenarios: Ground Truth (GT), comparing outputs to human references, and UQDB (User Question + DB Result), a reference-less approach vital for real-time production monitoring.
The NLR-BIRD Dataset
The study introduces NLR-BIRD, the first dedicated benchmark for NLR generation. Derived from the BIRD (Big Bench for Large-Scale Database Grounded Text-to-SQLs) dataset, it contains 1,468 samples across 11 domains (finance, sports, schools, etc.), including human-authored ground truth narrations for result sets where the sum of rows and columns is less than 500.
Performance and Benchmarking
Evaluation across 15 different judge models demonstrates that Combo-Eval consistently outperforms standalone metrics and matches or exceeds the accuracy of full LLM-based evaluation while significantly reducing computational overhead.
Key results include:
- Cost Efficiency: Combo-Eval reduced LLM judge calls by 61.43% in the GT scenario and 24.48% in the reference-less (UQDB) scenario.
- Superior Alignment: The framework achieved higher F1 scores and better agreement with human evaluators than isolated metrics or LLM-judges used alone.
- Model Scalability: Even smaller models like Grok-3-mini and GPT-4o-mini showed significant performance gains when integrated into the Combo-Eval framework.
Key Insights and Error Analysis
The authors identified that incomplete information is the primary failure mode in NLR generation, accounting for 64% of errors. Models frequently omit rows or columns when translating tabular data to text. Furthermore, the performance of even top-tier models like GPT-4o degrades significantly as table size increases, dropping from ~91% accuracy for small tables (3-9 cells) to ~30% for those with 50-499 cells.
Conclusion
Combo-Eval provides a scalable, accurate solution for evaluating NLR in Text-to-SQL systems. By leveraging the strengths of both traditional metrics and LLMs, it enables robust production monitoring and benchmarking. Future work will focus on extending these methods to "intelligent summarization" for massive datasets and using these evaluation insights to improve the generation models themselves.
Summary updated