Mind the Query: A Benchmark Dataset towards Text2Cypher Task

Vashu Chauhan, Shobhit Raj, Shashank Mujumdar, Avirup Saha, Anannay JainEMNLP 2025

TL;DR

This paper introduces "Mind the Query," a comprehensive benchmark dataset for the challenging Text2Cypher task, which translates natural language questions into executable Cypher queries for graph databases. By addressing limitations of existing datasets, this new resource aims to accelerate research in building more accessible graph database interfaces.

What Problem Does This Paper Address?

Graph databases are powerful for representing complex, interconnected data, but querying them often requires specialized knowledge of languages like Cypher. The Text2Cypher task aims to bridge this gap by allowing users to query these databases using natural language. However, existing datasets for training and evaluating Text2Cypher models suffer from several issues: many queries lack human verification, the associated graph databases are often missing (making evaluation difficult), and datasets are frequently limited to a single domain, hindering generalizability. This paper tackles these limitations by creating a high-quality, multi-domain benchmark dataset with available graph databases and rigorous validation.

Methods

The authors developed an end-to-end framework for generating the Text2Cypher dataset. This framework involves:

  • LLM-based NL-Cypher Pair Generation: Leveraging large language models (LLMs) to generate initial natural language questions and their corresponding Cypher queries. This process is guided by schema information, category-specific examples (few-shots), and detailed instructions.
  • Automatic Validation: A multi-stage pipeline to filter out incorrect or redundant pairs. This includes schema validation (checking if query elements match the graph schema), runtime validation (executing queries against the graph database), and value validation (ensuring literal values are correct).
  • Human-in-the-Loop Logical Verification: A crucial step where human annotators manually review queries for logical correctness, especially for complex query types. This also involves detailed annotation of failure cases to provide insights for model improvement.

The dataset generation process also incorporates a novel complexity-aware categorization scheme for queries, enabling more nuanced evaluation.

Datasets

The paper introduces the "Mind the Query" dataset, comprising 27,529 NL-Cypher query pairs across 11 real-world graph datasets. These datasets span diverse domains, including healthcare, geospatial data, and social media. The dataset includes the corresponding graph databases for each example, facilitating query execution and validation. This is a significant improvement over existing datasets that often lack this crucial component.

Key Insights and Ideas

  • The Importance of Human Verification: The authors highlight that LLM-generated queries, even if syntactically correct, often lack logical correctness without human oversight. Their rigorous manual validation process is key to ensuring dataset quality.
  • Multi-domain Coverage is Crucial: Restricting datasets to a single domain limits the generalizability of Text2Cypher models. The "Mind the Query" dataset's multi-domain nature is designed to promote more robust and adaptable models.
  • Complexity Matters: The paper introduces a categorization scheme for query complexity, revealing that models struggle significantly more with complex retrieval and aggregation queries compared to simple retrieval. This categorization is vital for fine-grained evaluation.
  • Fine-tuning Bridges the Gap: Even smaller models, when fine-tuned on this high-quality dataset, can achieve performance comparable to much larger, general-purpose models, demonstrating the dataset's value.

Key Results

  • The "Mind the Query" dataset contains 27,529 NL-Cypher pairs across 11 diverse graph datasets.
  • A rigorous validation pipeline, combining automated checks and human review, achieved a 97.2% logical correctness rate for automatically validated queries.
  • Experiments with various LLMs showed that DeepSeek-V3 and Llama-4-Maverick performed best in zero-shot settings.
  • Introducing few-shot examples improved performance by an average of ~2.5%.
  • Fine-tuning Llama-3.1-8B-Instruct on the dataset led to dramatic performance improvements, achieving 70.15% overall execution accuracy on the test set.
  • The study revealed a significant performance gap (~37% on average) between simple and complex query generation for LLMs.

Limitations and Future Work

The authors acknowledge that manual validation is time-intensive and ongoing, with initial efforts focused on complex query categories. They plan to extend manual validation to all categories. A known challenge is identifying logically correct queries that return no results, which is difficult to distinguish from true data absence. Future work aims to address this by decomposing complex queries and analyzing intermediate results. They also intentionally excluded closed-source models to prevent test-set leakage and ensure reproducibility, but plan to evaluate them in controlled settings in the future.

Summary updated

BibTeX
@inproceedings{chauhan2025mind,
  author = {Vashu Chauhan and Shobhit Raj and Shashank Mujumdar and Avirup Saha and Anannay Jain},
  title = {{Mind the Query: A Benchmark Dataset towards Text2Cypher Task}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure