TL;DR
This paper introduces MetaboT, a novel multi-agent LLM framework that translates natural language questions into SPARQL queries for metabolomics knowledge graphs. By decomposing the task into specialized agents and grounding them in authoritative data sources, MetaboT significantly improves accuracy and reduces hallucinations compared to single-LLM approaches, making complex data analysis more accessible to researchers.
What Problem Does This Paper Address?
Mass spectrometry-based metabolomics generates vast amounts of complex, high-dimensional data that are challenging to integrate and interpret. While knowledge graphs (KGs) offer a way to unify this information, their practical use is hindered by the steep learning curve associated with specialized query languages like SPARQL and representation languages like RDF. Existing LLM-based solutions for natural language to SPARQL translation often suffer from hallucinations and schema-compliance issues, limiting their reliability for domain-specific tasks. This paper aims to bridge this gap by providing an accessible and accurate method for querying metabolomics KGs.
Methods
MetaboT employs a multi-agent LLM architecture orchestrated by LangChain and LangGraph. The framework breaks down the natural language to SPARQL translation process into several specialized agents:
- Entry Agent: Manages the conversation flow and determines if a query is new or a follow-up.
- Validator Agent: Checks the query's scope and relevance against the KG schema.
- Supervisor Agent: Acts as the central coordinator, routing queries to appropriate agents.
- KG Agent: Resolves entities (e.g., chemical names, taxa) to their authoritative identifiers (IRIs) using specialized tools that query external resources like Wikidata and ChEMBL. This is a key differentiator, as it avoids relying solely on the LLM's parametric knowledge.
- SPARQL Query Runner Agent: Constructs and executes the SPARQL query using the resolved entities and schema information. It incorporates a refinement loop to handle query construction errors.
- Interpreter Agent: Processes the SPARQL results and generates user-friendly summaries or visualizations.
The system integrates eleven specialized tools for tasks like entity resolution, schema validation, and query execution.
Datasets
The primary dataset used for evaluation is the Experimental Natural Products Knowledge Graph (ENPKG), a comprehensive semantic graph integrating mass spectrometry data with taxonomic information, chemical structures, and biological activities of natural products.
The authors also constructed a synthetic validation dataset of 50 natural-language questions paired with manually verified reference SPARQL queries. These questions were drafted by domain experts and categorized by complexity (low, medium, high) to thoroughly test MetaboT's performance. This dataset is made publicly available.
Key Insights and Ideas
- Decomposition is Key: Breaking down the complex task of natural language to SPARQL translation into specialized agents significantly improves accuracy and robustness. Each agent has a clear, single responsibility.
- Authoritative Entity Resolution: Grounding entity resolution in external, authoritative resources (Wikidata, ChEMBL, etc.) is crucial for mitigating LLM hallucinations and ensuring schema compliance. This is a significant improvement over relying solely on LLM knowledge.
- Iterative Refinement for Robustness: The inclusion of a query refinement loop helps to distinguish between query construction errors and genuine data absence, leading to more reliable results.
- Accessibility for Domain Scientists: MetaboT aims to democratize access to complex metabolomics data by allowing researchers to query KGs using natural language, eliminating the need for specialized programming expertise.
Key Results
MetaboT, using GPT-40 within its multi-agent framework, achieved an 83.67% overall accuracy on the 49-question evaluation set. This is a substantial improvement compared to single-shot GPT-40 (8.16%) and MetaboT with GPT-40 mini (12.24%). Notably, MetaboT maintained 78.95% accuracy on high-complexity queries, where standalone GPT-40 failed completely. The primary error types observed were incorrect SPARQL syntax and validator misclassifications.
Limitations and Future Work
- Hallucinations: While significantly reduced, LLMs can still generate incorrect objects or properties, leading to errors in query construction or inappropriate tool selection.
- Non-determinism: The inherent variability of LLM outputs means identical queries might produce slightly different SPARQL commands, though prompt engineering and deterministic tools help mitigate this.
- Model Dependence and Cost: MetaboT currently requires a high-capacity LLM for optimal performance, leading to higher computational costs and potential barriers for some research communities. Prompt tuning may also be needed for different LLM backends.
- Single-Graph Querying: The current version is limited to querying a single knowledge graph. Future work aims to develop federated SPARQL query capabilities to query multiple KGs simultaneously.
- Evaluation Scope: The current evaluation focuses on SPARQL query generation, with downstream interpretation accuracy left for future work.
Future directions include evolving MetaboT into a broader toolbox for mass spectrometry data analysis, integrating more agents and components for common computational metabolomics tasks, and expanding interoperability with other tools and knowledge graphs. The MetaboLinkAI program aims to further these advancements.
Summary updated