TL;DR
This paper introduces pRAGe, a novel pipeline for generating medical paraphrases using small language models (SLMs) enhanced with retrieval-augmented generation (RAG). It addresses the critical need for accurate, understandable medical information for laypeople, demonstrating that SLMs, when augmented with external knowledge, can produce reliable paraphrases, mitigating the hallucination risks of larger models.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly accessible, but their use in medical contexts is fraught with peril due to "hallucinations" – generating incorrect or factually inaccurate information. This poses a significant risk to patient safety, especially when recommendations or prognoses are involved. Furthermore, the immense computational resources required by state-of-the-art LLMs make them inaccessible for many researchers and institutions. The paper tackles the challenge of generating accurate, scientifically grounded medical paraphrases that are easily understandable by laypeople, specifically focusing on sub-sentential paraphrases (short phrases or definitions) for medical terms. This is crucial because medical jargon often hinders effective communication between healthcare professionals and patients.
Methods
The core of the paper is the pRAGe (pipeline for Retrieval Augmented Generation and evaluation) system. pRAGe employs an encoder-retriever-decoder framework.
- Encoder: It uses French language models (like CamemBERT and DrBERT) to encode the input query and a knowledge base.
- Retriever: It retrieves relevant information from an external knowledge base.
- Decoder: It generates the simplified paraphrase or explanation.
The authors investigate the effectiveness of Small Language Models (SLMs) (1-7B parameters) in both vanilla inference (relying solely on the model's parametric knowledge) and within the pRAGe pipeline (augmented with external knowledge). They explore two key aspects:
- Quantization vs. Fine-tuning: Comparing the performance of quantized SLMs with fine-tuned models.
- Finetuning vs. Prompting: Analyzing the impact of finetuning SLMs on a medical paraphrase dataset versus using prompt engineering.
They also introduce RefoMed-KB, a French medical knowledge base derived from Wikipedia articles, to support the RAG system.
Datasets
- RefoMed: A unique, open-source dataset of medical sub-sentential paraphrases in French. It contains 6,297 pairs of medical terms and their corresponding paraphrases, extracted from scientific and simplified medical texts. This dataset was used for finetuning SLMs and for evaluating the RAG system.
- RefoMed-KB: A knowledge base built from the validation and testing sets of RefoMed, by extracting the top 3 Wikipedia articles for each medical term.
Key Insights and Ideas
- SLMs can be effective with RAG: The study demonstrates that even smaller, more accessible language models can achieve high-quality medical paraphrasing when augmented with external knowledge through a RAG pipeline.
- Finetuning improves medical accuracy: Finetuning SLMs on the RefoMed dataset significantly improves the correctness and conciseness of medical paraphrases, especially for short answers.
- pRAGe mitigates hallucinations: The RAG approach inherently reduces hallucinations by grounding responses in retrieved factual information, a critical advantage for medical applications.
- French medical SLMs are lacking: The paper highlights a scarcity of open-source, French-language SLMs specifically trained for medical NLG tasks, underscoring the importance of their work.
- Readability vs. Completeness Trade-off: While finetuning can improve correctness and completeness, it might sometimes lead to a slight decrease in readability for laypeople due to the increased use of medical terms.
Key Results
- BIOMISTRAL excels: The BIOMISTRAL model, particularly when fine-tuned and integrated into the pRAGe pipeline, showed the best performance in terms of strict correctness for short answers (90%).
- Finetuning boosts performance: Finetuning improved performance across various metrics, with a notable +23% improvement in strict completeness for the CamemBERT BIOMISTRAL pRAGe model in the 25-token setting.
- pRAGe vs. Base SLM: While base BIOMISTRAL performed well in zero-shot settings, pRAGe setups generally showed better correctness and completeness, especially when finetuned.
- Hallucination rates: BIOMISTRAL had low hallucination rates (4-6% in zero-shot), while BARThez, a summarization model, struggled with Q&A tasks and had higher hallucination rates (50%) in the pRAGe setting.
Limitations and Future Work
- Limited French SLM availability: The authors note the lack of specialized French medical NLG models.
- Readability challenges: While correctness and completeness improved with finetuning, readability for laypeople can be a challenge, and further research is needed to balance these aspects.
- Future work: The authors plan to test other French language models (Vigogne, Claire), conduct extensive annotation campaigns with medical specialists to ensure fair inter-annotator agreement on correctness, and further explore the impact of different knowledge bases. They also aim to investigate how generation is affected by the choice of knowledge base.
Summary updated