Utilizing Large Language Models for Natural Interface to Pharmacology Databases

Hong Lu, Chuan Li, Yinheng Li, Jie ZhaoarXiv 2023 · DOI · arXiv

Abstract

Pharmacology relies on vast, structured biological data repositories, yet accessing this information often requires specialized SQL knowledge. This paper introduces a natural language interface developed by Microsoft researchers that enables pharmacologists to query complex databases using plain English. By leveraging GPT-4 within a specialized framework featuring Named Entity Recognition (NER) and self-correction mechanisms, the system achieves high accuracy on expert-level queries, effectively lowering the technical barrier to data-driven drug discovery.

Introduction

The field of pharmacology is built upon a staggering volume of structured data, including information on genes, proteins, drugs, and diseases. Databases like PrimeKG aggregate this information, but querying them typically requires writing complex SQL or possessing deep knowledge of the underlying schema. This "SQL barrier" significantly slows down the pace of drug discovery. While Large Language Models (LLMs) show promise in code generation, they often struggle with the highly specific terminology and complex relational structures inherent in pharmacology. This research addresses the challenge of creating a reliable natural language interface that translates scientific questions into accurate database queries.

Methodology

The proposed framework consists of a dual-pipeline architecture designed to handle both question answering and data scarcity.

Question Answering (QA) Pipeline

The QA pipeline processes user queries through several stages:

  1. NER & Entity Linking: An in-house Named Entity Recognition model identifies biological entities, which are then linked to the database via Microsoft Cognitive Search and replaced with type-aware placeholders (e.g., [DRUG_NAME]).
  2. GPT-4 SQL Generation: The system utilizes GPT-4 to translate the processed question into SQL, employing advanced prompting techniques such as Chain-of-Thought (CoT), Question Decomposition, and Schema Linking.
  3. Self-Correction: The model reviews its generated SQL to identify and rectify errors before execution.

Question Generation (QG) Pipeline

To address the lack of specialized training data, the authors developed a QG pipeline. This pipeline uses an LLM to generate a synthetic dataset of questions based on the Knowledge Graph, providing both evaluation benchmarks and few-shot examples for the QA system.

Overview of the Question Generation and Question Answering pipelines.

Experimental Setup

The system was evaluated using three primary data sources:

  • PrimeKG: A comprehensive knowledge graph containing over 8 million edges and 30 relation types across 20 biomedical sources.
  • Synthetic Dataset: 264 questions (60 single-hop and 204 two-hop) generated by the system and validated by domain experts.
  • Realistic Dataset: 50 high-quality question-answer pairs curated by experts unfamiliar with the database schema, representing real-world use cases.

Results and Discussion

The evaluation revealed several key insights into the performance of LLMs in the pharmacological domain.

Model Performance and Ablation

GPT-4 demonstrated a significant performance advantage over ChatGPT. On the realistic dataset, the GPT-4 powered system achieved an F1-score of 0.712 and an Exact Match (EM) of 0.708. Ablation studies highlighted the critical role of specific components. For instance, while self-correction was vital for ChatGPT's performance, GPT-4 often generated correct SQL on the first attempt.

Performance comparison between ChatGPT and GPT-4 across different system configurations on the realistic dataset.

The NER Bottleneck

A primary finding was that Named Entity Recognition is the system's main bottleneck. When "Oracle" NER (manually provided correct entities) was used, performance improved significantly. For example, in synthetic tests, GPT-4's overall F1-score rose from 0.350 to 0.543 when the NER hurdle was removed.

Complexity Scaling

The system faced greater challenges with "two-hop" queries (reasoning across multiple relationships) compared to single-hop lookups. On synthetic data, GPT-4's F1-score dropped from 0.496 for single-hop questions to 0.343 for two-hop questions.

Conclusion

This research demonstrates the potential of LLMs to democratize access to complex pharmacological databases. While the current system serves as a strong proof-of-concept, the robustness of the Named Entity Recognition component remains the primary limiting factor. Future work will focus on improving entity linking and enhancing the system's ability to handle complex, multi-hop reasoning chains that reflect the true complexity of biological systems.

Summary updated

BibTeX
@misc{lu2023utilizing,
  author = {Hong Lu and Chuan Li and Yinheng Li and Jie Zhao},
  title = {{Utilizing Large Language Models for Natural Interface to Pharmacology Databases}},
  year = {2023},
  eprint = {2307.15717},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.15717},
}
Figure