Abstract
In the current paradigm of image captioning, deep learning models are trained to generate text from image embeddings of latent features. We challenge the assumption that these latent features ought to be high-dimensional vectors which require model fine tuning to handle. Here we propose Label Boosted Retrieval Augmented Generation (LaB-RAG), a text-based approach to image captioning that leverages image descriptors in the form of categorical labels to boost standard retrieval augmented generation (RAG) with pretrained large language models (LLMs). We study our method in the context of radiology report generation (RRG), where the task is to generate a clinician's report detailing their observations from a set of radiological images, such as X-rays. We argue that simple linear classifiers over extracted image embeddings can effectively transform X-rays into text-space as radiology-specific labels. In combination with standard RAG, we show that these derived text labels can be used with general-domain LLMs to generate radiology reports. Without ever training our generative language model or image feature encoder models, and without ever directly "showing" the LLM an X-ray, we demonstrate that LaB-RAG achieves better results across natural language and radiology language metrics compared with other retrieval-based RRG methods, while attaining competitive results compared to other fine-tuned vision-language RRG models. We further present results of our experiments with various components of LaB-RAG to better understand our method. Finally, we critique the use of a popular RRG metric, arguing it is possible to artificially inflate its results without true data-leakage.
Introduction
Automated radiology report generation (RRG) is a critical task at the intersection of computer vision and natural language processing. The prevailing research trend emphasizes the development of massive, end-to-end vision-language models through extensive supervised fine-tuning (SFT). While effective, this approach is computationally intensive, requires specialized hardware, and results in "black box" models that are difficult to adapt to new clinical settings or update with new knowledge.
Standard Retrieval Augmented Generation (RAG) offers a modular alternative by leveraging past cases to inform new generations. However, standard RAG often struggles in medical imaging because visual similarity in latent space does not always align with clinical similarity. To address this, we introduce LaB-RAG, a framework that uses clinical labels as a text-space bridge to enhance the retrieval and generation process.
Methods
LaB-RAG enhances the standard RAG pipeline through a modular "Label Boosting" approach. The process involves three primary stages:
- Image Embedding & Classification: We utilize frozen, domain-specific encoders (e.g., BioViL-T) to generate image embeddings. Instead of fine-tuning these encoders, we train simple logistic regression classifiers to predict 14 standard radiology labels from the embeddings.
- Label Boosted Filtering: During retrieval, the system does not rely solely on vector similarity. It filters or re-ranks retrieved reports to ensure they share the same clinical labels as the predicted labels of the query image.
- Label Boosted Formatting: The predicted labels are converted into natural language descriptors and inserted into the LLM prompt. This allows a general-domain LLM (e.g., Mistral-7B-Instruct) to "see" clinical findings through text.
The final report is generated using In-Context Learning (ICL), where the LLM generates text based on the retrieved examples and labels provided in the prompt, without any weight updates.
Results and Discussion
We evaluated LaB-RAG on the MIMIC-CXR and CheXpert Plus datasets using natural language metrics and clinical metrics such as F1-CheXbert and F1-RadGraph.
Key Findings
- Additive Gains: Our experiments demonstrate that both Label Filtering and Label Formatting provide independent and additive improvements to report quality. By combining these steps, LaB-RAG achieves a significant performance lift over standard RAG.
- State-of-the-Art Performance: On the MIMIC-CXR "Findings" section, LaB-RAG achieved state-of-the-art results on the F1-CheXbert metric, outperforming complex fine-tuned models like CheXagent and CXRMate.
- Efficiency and Modularity: LaB-RAG provides an 11% improvement in clinical accuracy over standard RAG with minimal additional computation. Its modular design allows for easy swapping of image encoders or LLMs.
While LaB-RAG excels in identifying clinical labels, it lags slightly behind some fine-tuned models on F1-RadGraph, which captures complex anatomical relationships. This suggests that while categorical labels are powerful, they may not fully capture the spatial nuances accessible to end-to-end models.
Conclusion
LaB-RAG demonstrates that high-quality, clinically accurate radiology reports can be generated without expensive end-to-end fine-tuning. By using simple classifiers to bridge the gap between visual features and text-space labels, we enable general-purpose LLMs to excel in specialized medical domains. Future work will explore the use of radiology-specific LLMs and more sophisticated label-based ranking logic.
Summary updated