Chase: A Large-Scale and Pragmatic Chinese Dataset for Cross-Database Context-Dependent Text-to-SQL

Jiaqi Guo, Ziliang Si, Yu Wang, Qian Liu, Ming Fan, Jian-Guang Lou, Zijiang Yang, Ting Liu

Abstract

CHASE is a large-scale, pragmatic Chinese dataset designed for cross-database context-dependent Text-to-SQL (XDTS). Existing benchmarks often suffer from high context-independence and simplicity biases. CHASE addresses these by employing intent-based recommendations and from-scratch creation (CHASE-C), resulting in more complex SQL and higher linguistic dependency. Evaluation of state-of-the-art models reveals a significant performance gap, with top systems achieving only 15.6% interaction accuracy, highlighting the need for more robust context-aware modeling in Chinese Text-to-SQL.

Introduction

The goal of Text-to-SQL is to build interfaces that allow users to query databases using natural language. The most challenging version of this task is Cross-Database Context-Dependent Text-to-SQL (XDTS). In this setting, systems must handle databases they were not exposed to during training and understand multi-turn conversations where later questions rely on earlier context (e.g., using pronouns or omitting previously mentioned entities).

Example of a multi-turn Chinese Text-to-SQL interaction in the CHASE dataset.

Despite progress, existing XDTS datasets like SParC and CoSQL suffer from two major biases: high context independence (where many questions can be answered without prior turns) and simplicity bias (where a large portion of queries are basic). CHASE was developed to overcome these limitations and provide a more rigorous testbed for Chinese language systems.

Addressing Dataset Biases

The authors identify that previous datasets often relied on "decomposing" complex queries into simpler steps. While this generates data, it naturally limits the difficulty of the resulting questions. Furthermore, up to 68% of questions in some existing benchmarks are context-independent, failing to truly test a model's ability to handle dialogue.

To build a more "pragmatic" dataset, CHASE introduces:

  • Intent Recommendation: Annotators were guided by randomly recommended "intents" (e.g., "Add a property" or "Add an ordering clause") to ensure diverse and cohesive conversation flows.
  • From-Scratch Creation (CHASE-C): 2,003 question sequences were created from scratch over 120 Chinese databases, avoiding the simplicity trap of decomposition.
  • Refined Translation (CHASE-T): A translated and manually corrected version of SParC, modified to be more coherent and context-heavy.

Linguistic Complexity and Dataset Analysis

CHASE consists of 5,459 coherent question sequences and 17,940 individual question-SQL pairs across 280 multi-table databases. It includes rich annotations for coreference, ellipsis, and schema linking.

Comparison of context dependency and linguistic phenomena across datasets.

As shown in Table 2, CHASE is significantly more conversational than its predecessors. Only 35.3% of its questions are context-independent, compared to 47.5% in SParC and 68.2% in CoSQL. This forces models to deal with heavy use of ellipsis (omitting words) and coreference (using pronouns). Additionally, the from-scratch creation method in CHASE-C increased the proportion of "Hard" and "Extra Hard" queries to over 44%, nearly double that of SParC.

Experimental Results

The authors evaluated three state-of-the-art (SOTA) models: EditSQL, IGSQL, and a modified RAT-SQL. The results indicate that CHASE is a significantly more difficult benchmark:

  • Performance Drop: While these models achieve ~60% accuracy on SParC, the best model (IGSQL) only managed 40.4% accuracy on individual questions in CHASE.
  • Sequence Failure: On Interaction Match (correctly predicting every SQL query in a multi-turn sequence), the best model plummeted to 15.6%.
  • The Chinese Challenge: Standard string-matching techniques for schema linking often fail in Chinese due to linguistic nuances (e.g., "song name" vs. "name of the song"). This suggests a need for more semantic matching strategies.

Conclusion

CHASE reveals that current SOTA models still have a long way to go in mastering pragmatic, multi-turn database querying in Chinese. By reducing simplicity and context-independence biases, CHASE provides a more realistic evaluation of XDTS systems. Future work must focus on improved schema linking and more robust context encoders to bridge the performance gap identified in this study.

Summary updated

BibTeX
@inproceedings{guo2021chase,
  author = {Jiaqi Guo and Ziliang Si and Yu Wang and Qian Liu and Ming Fan and Jian-Guang Lou and Zijiang Yang and Ting Liu},
  title = {{Chase: A Large-Scale and Pragmatic Chinese Dataset for Cross-Database Context-Dependent Text-to-SQL}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.180},
}
Figure