An Automated Evaluation Framework for Graph Database Query Generation Leveraging Large Language Models

Bailan He, Yushan Liu, Marcel Hildebrandt, Zifeng Ding, Yaomengxi Han, Volker TrespD2R2 2024 · OpenReview

Abstract

As Knowledge Graphs (KGs) become integral to industrial data management, the demand for Natural Language Interfaces (NLIs) has grown. This paper presents an automated pipeline for evaluating the translation of natural language into graph database queries (Cypher). By utilizing Large Language Models (LLMs) to generate synthetic benchmarks from query templates and employing execution-based metrics, the framework addresses the lack of standardized datasets in specialized domains like Supply Chain Management (SCM). Experiments with GPT-3.5 and GPT-4 demonstrate that schema-awareness and in-context learning significantly improve query accuracy, though an "execution gap" remains between textual similarity and functional correctness.

Introduction

In complex domains like Supply Chain Management, Knowledge Graphs allow for sophisticated data relationships. However, querying these graphs typically requires specialized knowledge of languages like Cypher. While LLMs offer the potential to act as "translators" for non-technical users, evaluating their performance is difficult. Standard text metrics like BLEU or ROUGE fail because functionally identical queries can have different syntax. Furthermore, manual evaluation by domain experts is slow and expensive. This research introduces a scalable, automated framework to bridge this gap.

The Automated Evaluation Pipeline

The proposed framework consists of a two-part pipeline designed to automate both dataset creation and model assessment:

  1. Synthetic Dataset Generation: The framework uses a "template-to-request" approach. It begins with query templates (e.g., MATCH (n)-[r]->(m:Label1)...), populates them with real-world SCM data to create "Gold Queries," and then uses an LLM to "back-translate" these queries into multiple natural language variations.
  2. Execution-Based Evaluation: To ensure functional accuracy, the framework employs three key metrics:
  3. Execution Rate (ER): The percentage of generated queries that run without syntax errors.
  4. Gold Query Accuracy (GQA): A semantic similarity measure using BERTScore to compare the generated query against the gold standard.
  5. Execution Accuracy (EA): The "ground truth" metric that compares the actual data results returned by both the generated and gold queries.
The automated pipeline for dataset creation and execution-based evaluation.

Experimental Setup and Prompting Strategies

The framework was tested using a real-world Siemens Supply Chain Knowledge Graph containing 65,277 nodes and 311,676 edges. The researchers produced a custom evaluation dataset of 825 validated query-request pairs covering categories like node matching, aggregation, and complex path traversals.

A central focus of the study was the impact of prompt engineering. The researchers compared three levels of prompt complexity:

  • Simple Prompt: Only the natural language request is provided.
  • Schema-aware Prompt: Includes the database schema (labels and relationship types) to ground the model.
  • In-Context Prompt: Provides the schema plus several few-shot examples of query-request pairs.
Comparison of the three prompting strategies: Simple, Schema-aware, and In-context.

Key Findings

The results highlight several critical insights for industrial LLM applications:

  • GPT-4 Superiority: GPT-4 consistently outperformed GPT-3.5 across all metrics. With 5-shot in-context learning, GPT-4 achieved a 93% Execution Rate and a Gold Query Accuracy of 0.75.
  • The Necessity of Schema: Without explicit schema information, models frequently suffer from "schema hallucination," guessing relationship names (e.g., using associated_with instead of the required related_to). Providing the schema led to a significant jump in GQA for GPT-3.5 from 0.47 to 0.62.
  • The Execution Gap: Execution Accuracy (EA) scores were generally lower than Gold Query Accuracy (GQA). This indicates that queries which appear semantically correct may still contain subtle logical errors that lead to incorrect data retrieval, emphasizing the need for execution-based testing over text-based metrics.

Conclusion

This framework provides a robust, automated method for benchmarking LLMs in the specialized task of graph query generation. By leveraging LLMs to build their own benchmarks and focusing on execution results, the research offers a path toward more reliable Natural Language Interfaces in industrial settings. Future work will explore multi-round dialogues and the performance of open-source models like Llama or Gemini.

Summary updated

BibTeX
@inproceedings{he2024automated,
  author = {Bailan He and Yushan Liu and Marcel Hildebrandt and Zifeng Ding and Yaomengxi Han and Volker Tresp},
  title = {{An Automated Evaluation Framework for Graph Database Query Generation Leveraging Large Language Models}},
  booktitle = {D2R2},
  year = {2024},
  url = {https://openreview.net/forum?id=oozpgrdQuP},
}
Figure