Abstract
Researchers at IBM have developed an LLM-powered pipeline that translates natural language requirements into precise GraphQL queries. By combining schema simplification, automated enrichment with natural language descriptions, and few-shot in-context learning using IBM’s Granite models, the system achieves up to 88% accuracy in retrieving data from complex, multi-source environments.
Introduction
While GraphQL is a powerful alternative to REST APIs—offering flexible data retrieval without the issues of over-fetching or under-fetching—it presents a high barrier to entry. Developers must have deep technical knowledge of specific, often massive, GraphQL schemas to write valid queries.
For Large Language Models (LLMs), the challenges are twofold:
- Context Limits: GraphQL schemas can be enormous, often exceeding the token limits of an LLM prompt.
- Hallucinations: LLMs frequently "hallucinate" field or table names that do not exist in the schema, leading to broken queries.
This paper describes an end-to-end architecture designed to bridge this gap, allowing users to query databases and REST APIs using natural language without needing to understand the underlying schema complexity.
Methodology
The proposed architecture focuses on optimizing the schema representation for the LLM rather than relying solely on model fine-tuning.
Key components of the pipeline include:
- Schema Enrichment: The system enriches the raw GraphQL schema by annotating fields with natural language descriptions. This provides the LLM with semantic "cues" to better map user intent to specific data fields.
- Schema Simplification: To fit within context windows and reduce noise, the system uses string parsing to strip away unnecessary implementation details (like resolver functions) from the schema before it is sent to the LLM.
- Prompt Engineering: The system utilizes a modular prompt structure consisting of high-level instructions, dynamic few-shot selection (In-Context Learning), and the simplified schema paired with the user's natural language query.
- Model Integration: The system utilizes IBM Granite Foundation Models (specifically the 20B-Code Base model) and the StepZen GraphQL engine to treat databases and REST APIs uniformly.
Experimental Setup
The study utilized a synthetically generated dataset comprising natural language utterances, corresponding GraphQL queries, and multi-source schemas. This approach was taken because existing public datasets often lack the multi-schema complexity required for real-world enterprise demonstrations.
Key Results and Insights
The researchers evaluated different prompting strategies on the IBM Granite 20B-Code Base Model. The results demonstrate that schema enrichment (comments) combined with relevant few-shot examples is more effective than simply increasing the number of examples.
| Prompt Strategy | Accuracy (%) |
|---|---|
| Fixed one-shot | 62.79% |
| Relevant one-shot | 72.09% |
| Relevant 5 shots | 81.40% |
| Relevant 20 shots | 84.00% |
| Relevant 5 shots + schema comments | 88.00% |
Key insights include:
- Comments Matter: The most significant performance jump came from adding natural language comments to the schema, suggesting LLMs rely heavily on semantic metadata to disambiguate similar-looking fields.
- Relevance over Quantity: Moving from a "fixed" one-shot example to "relevant" few-shot examples (where examples are chosen based on the user's specific query) significantly improved accuracy from ~63% to over 81%.
- Abstraction is Key: By using a GraphQL engine as an intermediary, the system simplifies the LLM's task from writing code for different protocols to writing a single query language.
Limitations and Future Work
While promising, the evaluation was performed on synthetic data; real-world performance on legacy enterprise systems with idiosyncratic naming conventions remains to be tested. Additionally, extremely complex reasoning tasks and scaling to truly massive schemas may require more aggressive retrieval-augmented generation (RAG) strategies in the future.
Summary updated