Abstract
Most available semantic parsing datasets, comprising pairs of natural utterances and logical forms, were collected solely for the purpose of training and evaluation of natural language understanding systems. As a result, they do not contain the richness and variety of naturally-occurring utterances, where humans ask about data they need or are curious about. In this work, we release SEDE, a dataset with 12,023 pairs of utterances and SQL queries collected from real usage on the Stack Exchange website. We show that these pairs contain a variety of real-world challenges rarely reflected in other semantic parsing datasets, propose an evaluation metric based on partial query clause comparison (PCM-F1), and conduct experiments with strong baselines, showing a large performance gap on SEDE compared to other common datasets.
Introduction
Text-to-SQL research has largely matured on "academic" datasets like Spider or WikiSQL. In these benchmarks, questions are typically generated by researchers or crowd-workers to align perfectly with a database schema, ensuring every requirement is explicitly stated. However, in real-world scenarios, human intent is often messy and under-specified. A user might request "top users" without specifying the metric (reputation, post count, etc.), or use complex SQL features like window functions and parameters. This paper introduces SEDE to bridge the gap between academic benchmarks and real-world usage.
The SEDE Dataset
The authors curated the SEDE (Stack Exchange Data Explorer) dataset by scraping public logs from the Stack Exchange website. The final dataset consists of 12,023 high-quality pairs, with 1,714 examples manually verified and corrected by humans to serve as a gold standard for validation and testing.
As shown in Table 1, SEDE is significantly more diverse than existing benchmarks. It features over 10,000 unique SQL templates and a queries-per-template ratio of 1.1, indicating that almost every query in the dataset represents a unique logical structure rather than a repeated pattern.
Evaluation Metric: PCM-F1
Standard metrics like Exact Match (EM) are often too rigid for complex, real-world SQL, while Execution Accuracy can be unreliable when questions are under-specified. To address this, the authors propose Partial Component Match F1 (PCM-F1). This metric decomposes SQL queries into sub-trees (e.g., SELECT, WHERE, ORDER BY) and calculates an F1 score based on component-level matching, allowing for a more nuanced evaluation of model performance that rewards partial correctness.
Key Insights and Challenges
SEDE queries introduce several challenges absent from standard benchmarks:
- The Ambiguity Gap: 87% of SEDE samples involve under-specification or hidden assumptions, requiring models to infer context not explicitly present in the text.
- SQL Complexity: Queries frequently utilize parameters (e.g.,
##UserId##), window functions, and sophisticated date manipulations. - Diversity: The low queries-per-template ratio prevents models from relying on simple pattern matching seen in cleaner datasets.
Figure 1 provides qualitative examples of how these challenges manifest in model predictions, particularly in handling hidden assumptions and complex date logic.
Results
The authors evaluated the T5 (Text-to-Text Transfer Transformer) model, a state-of-the-art architecture for this task. While T5-Large achieves a high 86.3% PCM-F1 on the Spider validation set, its performance plummets to 50.6% PCM-F1 on SEDE. Furthermore, Exact Match scores on SEDE were near zero for all models, highlighting that real-world SQL is too varied for simple string-matching evaluations to be useful. Interestingly, providing the model with the database schema did not significantly improve results, suggesting that the primary bottleneck is the complexity of the natural language logic rather than simple schema mapping.
Conclusion
The SEDE dataset reveals a significant performance gap in current Text-to-SQL models when faced with naturally occurring data. The prevalence of under-specification suggests that future research should focus on interactive systems capable of clarifying user intent. By providing a more realistic benchmark, SEDE encourages the development of models that can handle the inherent ambiguity and complexity of real-world data interaction.
Summary updated