Abstract
Medical Information Retrieval (MIR) is critical for clinical decision-making and patient education, yet it suffers from a severe lack of high-quality annotated data. This paper introduces AutoMIR, a framework featuring SL-HyDE (Self-Learning Hypothetical Document Embeddings), which enables Large Language Models (LLMs) and retrievers to mutually improve through unlabeled medical corpora. Furthermore, we present the Chinese Medical Information Retrieval Benchmark (CMIRB), the first comprehensive evaluation suite for this domain. Our results demonstrate that SL-HyDE significantly outperforms existing zero-shot methods across diverse medical tasks.
Introduction: The Medical Search Challenge
Building effective search engines for the medical domain faces three primary hurdles:
- Data Scarcity: Expert-level relevance labels are expensive and rare due to privacy concerns and the need for specialized knowledge.
- Knowledge Gaps: General LLMs often lack specialized depth, leading to hallucinations when generating domain-specific content.
- Semantic Mismatch: Standard embedding models struggle with dense medical jargon and scientific papers.
While methods like HyDE (Hypothetical Document Embeddings) attempt to solve this by using LLMs to generate "fake" documents for retrieval, their effectiveness is limited by the LLM's domain knowledge. AutoMIR addresses this by grounding the generation process in real-world unlabeled data.
Methods: SL-HyDE Framework
The core of AutoMIR is SL-HyDE, a self-learning pipeline that creates a "virtuous cycle" between a generator and a retriever.
The process involves two main stages:
- Self-Learning Generator: An LLM is tasked with reverse-engineering queries from real medical documents. It generates multiple hypothetical answers, which are then ranked by a retriever. The answer that most effectively helps the retriever identify the original document is used to fine-tune the LLM, ensuring it produces "retrievable" and accurate content.
- Self-Learning Retriever: Using the improved LLM, high-quality pseudo-documents are generated for the entire corpus. The retriever is then fine-tuned via contrastive learning, aligning the vector space of queries and pseudo-documents with actual relevant documents.
The CMIRB Benchmark
To evaluate MIR systems, the authors introduced the Chinese Medical Information Retrieval Benchmark (CMIRB). It comprises 10 datasets across five distinct tasks, covering the breadth of medical information needs.
The tasks include:
- Knowledge Retrieval: Finding facts in textbooks or encyclopedias (e.g., MedExam, DuBaike).
- Consultation Retrieval: Matching patient symptoms to doctor responses (e.g., MedicalRetrieval).
- News & Post Retrieval: Searching COVID-19 news and medical forum posts.
- Literature Retrieval: Navigating scientific abstracts and citations (e.g., CSLRel).
Key Results and Insights
SL-HyDE demonstrates robust performance across various architectures:
- Accuracy Gains: It achieved an average improvement of 4.87% in NDCG@10 over standard HyDE.
- Domain Adaptation: On the DuBaike dataset, performance surged by 15.61%, highlighting the framework's ability to adapt to specific encyclopedic styles.
- Model Agnostic: The method proved effective regardless of the underlying LLM (Qwen2, Llama2, ChatGLM3) or retriever (BGE, PEG, mE5).
A key insight is the role of the Retriever as a Teacher. By using the retriever's ranking to filter training data, SL-HyDE bypasses the need for human experts. This aligns the "hallucinations" of the LLM with the "indexing logic" of the search engine, creating a more cohesive retrieval system.
Conclusion
AutoMIR provides a scalable solution for medical search in zero-shot environments. While currently focused on the medical domain, the self-learning principles of SL-HyDE hold promise for other data-scarce fields like law or finance. Future work will explore more complex multi-step reasoning through reinforcement learning.
Summary updated