Abstract
Pharmacology relies on vast, structured biological data repositories, yet accessing this information often requires specialized SQL knowledge. This paper introduces a natural language interface developed by Microsoft researchers that enables pharmacologists to query complex databases using plain English. By leveraging GPT-4 within a specialized framework featuring Named Entity Recognition (NER) and self-correction mechanisms, the system achieves high accuracy on expert-level queries, effectively lowering the technical barrier to data-driven drug discovery.
Introduction
The field of pharmacology is built upon a staggering volume of structured data, including information on genes, proteins, drugs, and diseases. Databases like PrimeKG aggregate this information, but querying them typically requires writing complex SQL or possessing deep knowledge of the underlying schema. This "SQL barrier" significantly slows down the pace of drug discovery. While Large Language Models (LLMs) show promise in code generation, they often struggle with the highly specific terminology and complex relational structures inherent in pharmacology. This research addresses the challenge of creating a reliable natural language interface that translates scientific questions into accurate database queries.
Methodology
The proposed framework consists of a dual-pipeline architecture designed to handle both question answering and data scarcity.
Question Answering (QA) Pipeline
The QA pipeline processes user queries through several stages:
- NER & Entity Linking: An in-house Named Entity Recognition model identifies biological entities, which are then linked to the database via Microsoft Cognitive Search and replaced with type-aware placeholders (e.g.,
[DRUG_NAME]). - GPT-4 SQL Generation: The system utilizes GPT-4 to translate the processed question into SQL, employing advanced prompting techniques such as Chain-of-Thought (CoT), Question Decomposition, and Schema Linking.
- Self-Correction: The model reviews its generated SQL to identify and rectify errors before execution.
Question Generation (QG) Pipeline
To address the lack of specialized training data, the authors developed a QG pipeline. This pipeline uses an LLM to generate a synthetic dataset of questions based on the Knowledge Graph, providing both evaluation benchmarks and few-shot examples for the QA system.
Experimental Setup
The system was evaluated using three primary data sources:
- PrimeKG: A comprehensive knowledge graph containing over 8 million edges and 30 relation types across 20 biomedical sources.
- Synthetic Dataset: 264 questions (60 single-hop and 204 two-hop) generated by the system and validated by domain experts.
- Realistic Dataset: 50 high-quality question-answer pairs curated by experts unfamiliar with the database schema, representing real-world use cases.
Results and Discussion
The evaluation revealed several key insights into the performance of LLMs in the pharmacological domain.
Model Performance and Ablation
GPT-4 demonstrated a significant performance advantage over ChatGPT. On the realistic dataset, the GPT-4 powered system achieved an F1-score of 0.712 and an Exact Match (EM) of 0.708. Ablation studies highlighted the critical role of specific components. For instance, while self-correction was vital for ChatGPT's performance, GPT-4 often generated correct SQL on the first attempt.
The NER Bottleneck
A primary finding was that Named Entity Recognition is the system's main bottleneck. When "Oracle" NER (manually provided correct entities) was used, performance improved significantly. For example, in synthetic tests, GPT-4's overall F1-score rose from 0.350 to 0.543 when the NER hurdle was removed.
Complexity Scaling
The system faced greater challenges with "two-hop" queries (reasoning across multiple relationships) compared to single-hop lookups. On synthetic data, GPT-4's F1-score dropped from 0.496 for single-hop questions to 0.343 for two-hop questions.
Conclusion
This research demonstrates the potential of LLMs to democratize access to complex pharmacological databases. While the current system serves as a strong proof-of-concept, the robustness of the Named Entity Recognition component remains the primary limiting factor. Future work will focus on improving entity linking and enhancing the system's ability to handle complex, multi-hop reasoning chains that reflect the true complexity of biological systems.
Summary updated