TL;DR
This paper introduces a novel, large-scale, cross-domain dataset for Text-to-GraphQL query generation, addressing the critical lack of benchmarks for evaluating Large Language Models (LLMs) in this domain. The authors demonstrate that even state-of-the-art LLMs struggle significantly with this task, highlighting the need for specialized datasets to improve their performance.
What Problem Does This Paper Address?
GraphQL is a powerful API query language that allows clients to fetch precisely the data they need, minimizing over-fetching and improving efficiency. However, crafting complex GraphQL queries can be challenging, especially when dealing with intricate schemas and diverse data sources. While Large Language Models (LLMs) show promise in generating these queries from natural language, their performance is hampered by a lack of publicly available, diverse GraphQL schemas for training and evaluation. Existing LLMs often produce invalid or suboptimal queries due to limited exposure. Crucially, there's no comprehensive benchmark dataset to reliably assess and compare the capabilities of LLMs in this specific task.
Methods
The authors developed a novel dataset through a multi-stage process:
- Schema Curation: They gathered approximately 200 relational databases from diverse sources (CoSQL, TPCH, WikiSQL) and created custom APIs. These were then ingested into the IBM StepZen GraphQL engine to generate initial GraphQL schemas.
- Schema Enrichment: To increase complexity and cover a wider range of GraphQL features, these basic schemas were manually edited, merged, and enriched with constructs like interfaces, input types, filters, and more.
- Query Generation: Two primary methods were used:
- Automated Generation: A rule-based GraphQL query generator was used to create basic projection queries, which were then enriched with natural language descriptions. This contributed about 20% of the dataset.
- Manual Creation: Researchers manually crafted GraphQL queries and their corresponding natural language descriptions using the StepZen UI. This more labor-intensive approach accounted for about 80% of the dataset and ensured higher quality and complexity.
- Dataset Enhancement: The IBM Granite-20B-Code-Instruct LLM was used to generate multiple paraphrases for each natural language query, increasing linguistic variety.
- Validation: A custom web-based user interface was developed for systematic review and annotation of each data element (schema, query, NL description) by six researchers, requiring approximately 1200 person-hours.
Datasets
The paper introduces a new, large-scale, cross-domain Text-to-GraphQL query operation dataset.
- Training Dataset: Contains 10,940 training triples, spanning 185 cross-source data stores.
- Test Dataset: Comprises 957 test data triples, covering 14 cross-source data stores. This dataset uses primarily different schemas than the training set to ensure generalizability.
The dataset is categorized into six overlapping types: Hops, Filters, Alias, Fragments, Multi Type, and Multiple-Endpoints, to capture various GraphQL complexities.
Key Insights and Ideas
- LLMs Struggle with GraphQL: Even the best-performing LLMs achieved only around 50% accuracy on the test dataset with a few in-context examples, indicating a significant gap in their current capabilities for this task.
- Need for Specialized Data: The limited performance of LLMs underscores the critical need for large, diverse, and complex datasets like the one presented to enable effective fine-tuning and benchmarking.
- Complexity of GraphQL: The dataset is designed to capture intricate GraphQL features (e.g., multiple hops, complex filters, aliases, fragments) that pose challenges for automated generation.
- In-Context Learning Limitations: While providing in-context examples helps, LLM context window limitations can be a bottleneck, especially with large GraphQL schemas, making fine-tuning a more viable path.
Key Results
- The best-performing LLM achieved approximately 50% accuracy on the test dataset with one in-context example.
- In the zero-shot setting, most LLMs achieved less than 15% overall accuracy, highlighting insufficient exposure to GraphQL data during pre-training.
- The introduction of one or two in-context examples led to marginal performance improvements in some models.
- Models with lower context limits faced "length errors" due to prompts exceeding their token limits, especially with large schemas.
Limitations and Future Work
- Schema Compatibility: The dataset is compatible with the StepZen GraphQL Engine, but transformation scripts may be needed to adapt schemas and queries for other engines.
- Future Work: The authors plan to expand the dataset to relax current assumptions and address limitations. They also intend to develop scripts for adapting the dataset to various GraphQL engines and further expand its scope. They suggest strategies like schema filtering, finding more relevant in-context examples, dynamic adjustment of in-context examples, and lightweight PEFT tuning (like prompt tuning) to improve LLM performance.
Summary updated