TL;DR
This paper introduces ZOGRASCOPE, a novel benchmark for semantic parsing over property graphs (PGs) using Cypher queries. It addresses the scarcity of PG-specific resources by providing a manually annotated dataset derived from a real-world crime investigation graph, enabling better evaluation of natural language interfaces for PGs.
What Problem Does This Paper Address?
Natural language interfaces (NLIs) are crucial for making complex data accessible, and knowledge graphs (KGs) are increasingly used to represent this data. Property graphs (PGs), with their flexible schema and explicit property distinction, are gaining traction in industry. However, existing benchmarks for semantic parsing (translating natural language to queries) primarily focus on RDF-style graphs, leaving a gap for PGs. Creating high-quality, manually annotated datasets for PGs is expensive and labor-intensive, leading to reliance on synthetic or adapted benchmarks that may not fully capture PG nuances. This paper aims to fill this gap by introducing ZOGRASCOPE, a benchmark specifically designed for PGs and the Cypher query language.
Methods
The ZOGRASCOPE benchmark was created through a meticulous five-step process:
- Anonymized Query Generation: Variable anonymized Cypher queries were automatically generated based on a tree pattern approach, ensuring structural consistency with the underlying graph.
- Manual Natural Language Annotation: Expert annotators manually created natural language (NL) realizations for each anonymized Cypher query.
- Value Instantiation: Anonymized variables in both the Cypher queries and their NL counterparts were instantiated with valid graph data, ensuring queries returned non-empty results. Three instantiations were generated per query.
- LLM Paraphrasing: To enhance lexical diversity, a large language model (GPT-4o) was used to generate paraphrases of the instantiated NL questions.
- Human Verification and Correction: Human annotators, proficient in Cypher, reviewed and corrected all generated paraphrases, ensuring the final dataset's quality and accuracy.
The paper also details experiments evaluating various transformer-based models (Llama 3.2-3B, Qwen3 4B, Mistral 7B, and GPT-4o) under four learning settings: fine-tuning, dynamic in-context learning (ICL), fixed ICL, and zero-shot.
Datasets
The primary dataset introduced is ZOGRASCOPE. It is built upon the publicly available "Pole graph," a property graph representing crime investigations, containing 61,521 nodes and 105,840 edges, with 11 entity classes, 32 properties, and 17 relation types. ZOGRASCOPE comprises approximately 5,000 manually annotated pairs of natural language questions and Cypher queries. The dataset is organized into three partitions:
- IID (Independent and Identically Distributed): Test samples correspond to templates seen during training.
- Compositional: Test samples use templates not observed during training, testing generalization.
- Length: Queries are split based on their length (number of nodes), with training sets having 2-3 nodes and test sets having 4-5 nodes.
Key Insights and Ideas
- Manual Annotation is Key: The paper emphasizes the critical role of manual annotation by domain experts for creating a high-quality, realistic benchmark that truly reflects the nuances of property graphs and Cypher.
- Compositional Generalization is Hard: Performance significantly drops on the compositional partition, highlighting the challenge LLMs face in generalizing to unseen query structures.
- Length Matters: Longer queries are substantially more difficult for models to parse accurately, indicating a struggle with complex structural understanding.
- Supervision is Crucial: Performance scales with the amount of supervision available, with fine-tuning yielding the best results, followed by dynamic ICL, fixed ICL, and finally zero-shot.
- Larger Models Don't Always Win: While larger models generally perform better, fine-tuning smaller models can sometimes outperform larger models in zero-shot or ICL settings, suggesting that effective training strategies are paramount.
Key Results
- Fine-tuning Dominates: Fine-tuning consistently achieved the highest accuracy across all models and partitions, with GPT-4o reaching near-perfect performance on the iid partition.
- IID vs. Compositional: Models achieved high accuracy on the iid partition (e.g., GPT-4o ~96.74%), but performance dropped significantly on the compositional partition (e.g., GPT-4o ~66.79%).
- Length Challenge: The length partition proved the most challenging, with accuracies dropping considerably for longer queries.
- Zero-Shot Performance: Zero-shot performance was generally low across all models, underscoring the need for some form of supervision or examples.
- ICL Benefits: Even a few in-context examples (fixed or dynamic) substantially improved performance over zero-shot. Dynamic ICL generally outperformed fixed ICL.
- Model Size Correlation: Larger models like GPT-4o generally outperformed smaller models, but fine-tuning smaller models could be competitive.
Limitations and Future Work
The authors acknowledge several limitations:
- Query Complexity: The current benchmark does not include highly complex query patterns like nested sub-queries. Future work aims to extend the dataset to cover these.
- Entity Value Instantiation: While paraphrasing adds lexical variety, the core mapping of entity values might still be relatively straightforward.
- Language Specificity: The dataset is currently only in English. Future work could explore creating multilingual versions.
The authors express hope that ZOGRASCOPE will serve as a valuable resource for advancing research in semantic parsing over property graphs.
Summary updated