TL;DR
This paper explores how well current Large Language Models (LLMs), specifically GPT-4o, can retrieve information from PubChem, a massive biological database. While LLMs show promise for simplifying database access through natural language, they currently struggle with complex biological queries and often fail to adhere to specific database constraints, highlighting the need for more domain-specific LLM development.
What Problem Does This Paper Address?
Biological and biomedical research relies heavily on vast databases like PubChem, which store an immense amount of structured and unstructured data. However, accessing and retrieving specific information from these databases is often a manual, time-consuming, and complex process. This can be a significant bottleneck for researchers, especially those without extensive programming backgrounds. Large Language Models (LLMs) with search and code generation capabilities offer a potential solution by enabling natural language queries and automating data retrieval. This paper investigates the current capabilities and limitations of these "search-enabled" LLMs in tackling real-world biological data retrieval tasks from PubChem.
Methods
The researchers evaluated GPT-4o's performance on eight common PubChem retrieval tasks. They adapted existing PubChem protocols into prompts for the LLM, experimenting with both direct prompting and prompts designed to generate programmatic access (i.e., code to query PubChem's API). They compared the LLM's outputs to "gold answers" obtained through manual execution of the protocols on the PubChem web interface and "silver answers" obtained through programmatic access. Prompt engineering, including adding context and refining phrasing, was used to improve GPT-4o's responses.
Datasets
The study used the PubChem database as its primary data source. The eight retrieval protocols, adapted from a previous publication, served as the evaluation benchmarks. These protocols covered tasks such as identifying gene-protein interactions, finding similar compounds, retrieving bioactivity data, and locating stereoisomers and isotopomers. No new datasets were introduced.
Key Insights and Ideas
- LLMs as Tutors, Not Replacements: While LLMs can't fully replace manual search yet, they show potential as helpful "tutors" by guiding users through complex database interactions and explaining processes.
- Programmatic Access is Key: LLMs are significantly better at generating URLs for programmatic access to PubChem than directly retrieving information through natural language prompts.
- Domain Knowledge Gap: LLMs struggle with tasks requiring deep understanding of chemistry and biology, such as applying specific filtering rules (like Lipinski's rule of five) or handling complex chemical structures (like SMILES strings).
- External Data Leakage: LLMs often pull information from external sources (like Wikipedia) even when instructed to use only PubChem data, indicating a lack of strict adherence to constraints.
- Interactivity Advantage: The conversational nature of LLMs allows for iterative refinement and building on previous queries, which is beneficial for complex, multi-step searches.
Key Results
- Direct Prompting Limitations: For direct retrieval via natural language prompts, GPT-4o provided incomplete or incorrect answers for most of the eight protocols. In some cases, it even hallucinated data.
- Programmatic Access Success: When prompted to generate programmatic access URLs, GPT-4o was successful in 3 out of 5 cases where programmatic access was available, and these generated URLs often led to correct (silver) answers.
- Web Interface Superiority: In several instances, the PubChem web interface offered filtering capabilities not available through programmatic access, leading to differences between gold and silver answers.
- Variability in Responses: The LLM's responses showed sensitivity to prompt wording, leading to inconsistent results and challenges in reproducibility.
Limitations and Future Work
The authors acknowledge several limitations:
- Instruction Following: GPT-4o struggled to strictly follow instructions, often accessing external data and failing to apply complex filters or rules.
- Multi-modal Capabilities: The LLM could not perform searches using chemical drawings or SMILES strings directly.
- Code Execution: While GPT-4o can generate code, it cannot execute it, requiring manual evaluation of generated URLs.
- Domain Expertise: A significant lack of fundamental understanding of chemistry and biology hinders the LLM's ability to interpret and filter biological data effectively.
Future work should focus on:
- Enhancing Domain Understanding: Developing LLMs with deeper knowledge of chemistry, biology, and biochemistry.
- Integrating Analytical Tools: Providing LLMs with access to external analytical tools like RDKit and PubChemPy.
- Knowledge Graphs: Integrating knowledge graphs to better link chemical entities and their properties.
- Specialized Chatbots: Creating chatbots specifically designed for biological databases like PubChem, combining natural language ease with precise search capabilities.
- Task Decomposition: Enabling LLMs to break down complex queries into manageable sub-tasks.
Summary updated