A Retrieval-Augmented Knowledge Mining Method with Deep Thinking LLMs for Biomedical Research and Clinical Support

Yichun Feng, Jiawei Wang, Ruikun He, Lu Zhou, Yixue Li

Abstract: Knowledge graphs and large language models (LLMs) are key tools for biomedical knowledge integration and reasoning, facilitating structured organization of scientific articles and discovery of complex semantic relationships. However, current methods face challenges: knowledge graph construction is limited by complex terminology, data heterogeneity, and rapid knowledge evolution, while LLMs show limitations in retrieval and reasoning, making it difficult to uncover cross-document associations and reasoning pathways. To address these issues, we propose a pipeline that uses LLMs to construct a biomedical knowledge graph (BioStrataKG) from large-scale articles and builds a cross-document question-answering dataset (BioCDQA) to evaluate latent knowledge retrieval and multi-hop reasoning. We then introduce Integrated and Progressive Retrieval-Augmented Reasoning (IP-RAR) to enhance retrieval accuracy and knowledge reasoning. IP-RAR maximizes information recall through Integrated Reasoning-based Retrieval and refines knowledge via Progressive Reasoning-based Generation, using self-reflection to achieve deep thinking and precise contextual understanding. Experiments show that IP-RAR improves document retrieval F1 score by 20% and answer generation accuracy by 25% over existing methods. This framework helps doctors efficiently integrate treatment evidence for personalized medication plans and enables researchers to analyze advancements and research gaps, accelerating scientific discovery and decision-making.

Introduction

Biomedical research is currently characterized by an explosion of data, with millions of papers published annually. However, the knowledge within these documents remains fragmented, highly technical, and constantly evolving. For clinicians and researchers, "connecting the dots" between disparate findings—such as a gene identified in one study and a drug interaction described in another—is a monumental challenge.

Standard Retrieval-Augmented Generation (RAG) frameworks often struggle in this domain due to terminology complexity (synonyms and polysemous terms) and reasoning gaps. Specifically, standard models frequently fail at "multi-hop" reasoning, which requires synthesizing evidence from multiple documents to reach a valid conclusion.

Overview of the biomedical knowledge mining process, from sources to clinical applications.

The IP-RAR Framework

To overcome these limitations, we developed IP-RAR (Integrated and Progressive Retrieval-Augmented Reasoning). This framework utilizes a novel two-stage process to mine complex biomedical data:

  1. BioStrataKG (Stratified Knowledge Graph): A dual-layer knowledge base. The first layer maps relationships between entities (genes, proteins, drugs, diseases), while the second layer maps relationships between the documents themselves, such as shared methodologies, citations, and research directions.
  2. Integrated Reasoning-based Retrieval: Instead of simple keyword searches, the system employs DeepSeek-V3 to perform "pre-retrieval reasoning." It generates keywords and a "virtual answer" (a hypothesis) to provide a richer semantic target for the retrieval engine.
  3. Progressive Reasoning-based Generation: This stage incorporates a self-reflection mechanism to evaluate the relevance of retrieved document chunks. It then utilizes DeepSeek-R1—a model optimized for complex chain-of-thought reasoning—to synthesize the final answer based only on high-quality evidence.

Datasets and Methodology

The efficacy of IP-RAR was validated using a newly introduced dataset, BioCDQA, which contains 1,183 complex, cross-document question-answer pairs derived from over 68,000 biomedical papers. The study also utilized standard benchmarks including BioASQ for semantic indexing and MASH-QA for multi-span question answering.

Detailed pipeline for BioStrataKG construction and BioCDQA dataset characteristics.

Key Insights and Results

A critical innovation in this research is the use of "virtual answers" during retrieval. By generating a hypothetical answer before querying the database, the system creates a more effective semantic bridge to relevant documents than a raw user query could provide. Furthermore, the stratification of the knowledge graph allows the system to address both entity-level facts and broader research trends with equal precision.

Experimental results demonstrate that IP-RAR significantly outperforms popular baselines like SELF-RAG and CRAG:

  • Retrieval Performance: On the BioCDQA dataset, IP-RAR achieved an F1-score of 34.96%, an improvement of 12–18 percentage points over baselines.
  • Answer Accuracy: In GPT-4-based evaluations, IP-RAR scored 76.41%, a substantial jump from the 25–41% range seen in other RAG frameworks.
  • Document Ranking: The system achieved a 95.12% Mean Average Precision (MAP) in BioASQ document retrieval.

Conclusion and Future Work

IP-RAR represents a significant step forward in biomedical knowledge mining, offering a 25% boost in answer accuracy for complex queries. While currently focused on text, future work will aim to integrate multimodal data (images, charts, and tables) and explore agent-based interactions to further refine the "deep thinking" process. This framework holds the potential to accelerate scientific discovery and provide critical support for personalized clinical decision-making.

Summary updated

Abstract

Knowledge graphs and large language models (LLMs) are key tools for biomedical knowledge integration and reasoning, facilitating structured organization of scientific articles and discovery of complex semantic relationships. However, current methods face challenges: knowledge graph construction is limited by complex terminology, data heterogeneity, and rapid knowledge evolution, while LLMs show limitations in retrieval and reasoning, making it difficult to uncover cross-document associations and reasoning pathways. To address these issues, we propose a pipeline that uses LLMs to construct a biomedical knowledge graph (BioStrataKG) from large-scale articles and builds a cross-document question-answering dataset (BioCDQA) to evaluate latent knowledge retrieval and multi-hop reasoning. We then introduce Integrated and Progressive Retrieval-Augmented Reasoning (IP-RAR) to enhance retrieval accuracy and knowledge reasoning. IP-RAR maximizes information recall through Integrated Reasoning-based Retrieval and refines knowledge via Progressive Reasoning-based Generation, using self-reflection to achieve deep thinking and precise contextual understanding. Experiments show that IP-RAR improves document retrieval F1 score by 20\% and answer generation accuracy by 25\% over existing methods. This framework helps doctors efficiently integrate treatment evidence for personalized medication plans and enables researchers to analyze advancements and research gaps, accelerating scientific discovery and decision-making.
BibTeX
@inproceedings{feng2025retrieval,
  author = {Yichun Feng and Jiawei Wang and Ruikun He and Lu Zhou and Yixue Li},
  title = {{A Retrieval-Augmented Knowledge Mining Method with Deep Thinking LLMs for Biomedical Research and Clinical Support}},
  booktitle = {GigaScience},
  year = {2025},
  doi = {10.1093/gigascience/giaf109},
}
Figure