Abstract
CHASE is a large-scale, pragmatic Chinese dataset designed for cross-database context-dependent Text-to-SQL (XDTS). Existing benchmarks often suffer from high context-independence and simplicity biases. CHASE addresses these by employing intent-based recommendations and from-scratch creation (CHASE-C), resulting in more complex SQL and higher linguistic dependency. Evaluation of state-of-the-art models reveals a significant performance gap, with top systems achieving only 15.6% interaction accuracy, highlighting the need for more robust context-aware modeling in Chinese Text-to-SQL.
Introduction
The goal of Text-to-SQL is to build interfaces that allow users to query databases using natural language. The most challenging version of this task is Cross-Database Context-Dependent Text-to-SQL (XDTS). In this setting, systems must handle databases they were not exposed to during training and understand multi-turn conversations where later questions rely on earlier context (e.g., using pronouns or omitting previously mentioned entities).
Despite progress, existing XDTS datasets like SParC and CoSQL suffer from two major biases: high context independence (where many questions can be answered without prior turns) and simplicity bias (where a large portion of queries are basic). CHASE was developed to overcome these limitations and provide a more rigorous testbed for Chinese language systems.
Addressing Dataset Biases
The authors identify that previous datasets often relied on "decomposing" complex queries into simpler steps. While this generates data, it naturally limits the difficulty of the resulting questions. Furthermore, up to 68% of questions in some existing benchmarks are context-independent, failing to truly test a model's ability to handle dialogue.
To build a more "pragmatic" dataset, CHASE introduces:
- Intent Recommendation: Annotators were guided by randomly recommended "intents" (e.g., "Add a property" or "Add an ordering clause") to ensure diverse and cohesive conversation flows.
- From-Scratch Creation (CHASE-C): 2,003 question sequences were created from scratch over 120 Chinese databases, avoiding the simplicity trap of decomposition.
- Refined Translation (CHASE-T): A translated and manually corrected version of SParC, modified to be more coherent and context-heavy.
Linguistic Complexity and Dataset Analysis
CHASE consists of 5,459 coherent question sequences and 17,940 individual question-SQL pairs across 280 multi-table databases. It includes rich annotations for coreference, ellipsis, and schema linking.
As shown in Table 2, CHASE is significantly more conversational than its predecessors. Only 35.3% of its questions are context-independent, compared to 47.5% in SParC and 68.2% in CoSQL. This forces models to deal with heavy use of ellipsis (omitting words) and coreference (using pronouns). Additionally, the from-scratch creation method in CHASE-C increased the proportion of "Hard" and "Extra Hard" queries to over 44%, nearly double that of SParC.
Experimental Results
The authors evaluated three state-of-the-art (SOTA) models: EditSQL, IGSQL, and a modified RAT-SQL. The results indicate that CHASE is a significantly more difficult benchmark:
- Performance Drop: While these models achieve ~60% accuracy on SParC, the best model (IGSQL) only managed 40.4% accuracy on individual questions in CHASE.
- Sequence Failure: On Interaction Match (correctly predicting every SQL query in a multi-turn sequence), the best model plummeted to 15.6%.
- The Chinese Challenge: Standard string-matching techniques for schema linking often fail in Chinese due to linguistic nuances (e.g., "song name" vs. "name of the song"). This suggests a need for more semantic matching strategies.
Conclusion
CHASE reveals that current SOTA models still have a long way to go in mastering pragmatic, multi-turn database querying in Chinese. By reducing simplicity and context-independence biases, CHASE provides a more realistic evaluation of XDTS systems. Future work must focus on improved schema linking and more robust context encoders to bridge the performance gap identified in this study.
Summary updated