TL;DR
This paper introduces X-MIR, a novel framework for explainable medical image retrieval. By adapting saliency map techniques from classification to retrieval, X-MIR helps users understand why certain medical images are considered similar to a query, fostering trust and aiding clinical decision-making.
What Problem Does This Paper Address?
Deep learning models have achieved remarkable success in medical imaging, but their "black box" nature poses a significant barrier to adoption in high-stakes fields like healthcare. While explainable AI (XAI) techniques, particularly saliency maps, have been explored for classification tasks, their application to medical image retrieval – finding similar images in a large archive – remains less studied. This lack of interpretability means clinicians might not understand why a retrieval system suggests a particular image, hindering trust and potentially leading to misinterpretations. The paper aims to bridge this gap by making medical image retrieval systems more transparent.
Methods
The core of X-MIR lies in adapting existing saliency map techniques for image retrieval. Instead of explaining a classification decision, the goal is to highlight regions in a retrieved image that contribute most to its similarity with a query image. The authors explore three types of similarity-based saliency maps:
- Occlusion-based: A black-box approach where parts of the retrieved image are systematically occluded, and changes in similarity to the query image are measured. Regions whose occlusion significantly alters similarity are deemed important.
- Attention-based: This method leverages triplet loss to learn similarity attention, essentially showing where the model "looks" when comparing images.
- Activation Mapping: Similar to attention-based methods, this approach uses features from the last convolutional layer to compute point-wise similarity maps.
To quantitatively evaluate these saliency maps, the authors introduce novel causal metrics adapted from classification: insertion and deletion. These metrics measure how image similarity changes when pixels are added (insertion) or removed (deletion) from the retrieved image, guided by the saliency map. Higher insertion AUC and lower deletion AUC indicate better saliency maps.
Datasets
The authors evaluate their approach on two publicly available medical imaging datasets:
- COVIDx dataset: A large dataset of chest X-rays for COVID-19 detection, containing images of normal, pneumonia, and COVID-19 cases.
- ISIC 2017 skin lesion dataset: A dataset focused on skin lesion classification, with images of benign nevi, seborrheic keratosis, and melanoma.
Key Insights and Ideas
- Saliency maps are transferable to retrieval: The core idea that saliency maps, traditionally used for classification, can be effectively adapted to explain image similarity in retrieval tasks is a key contribution.
- Understanding retrieval failures: The paper demonstrates how saliency maps can reveal why a retrieval system might fail, by showing that incorrect retrievals often focus on irrelevant regions (e.g., shoulders instead of lungs in X-rays).
- Quantitative evaluation is crucial: The authors highlight the limitations of purely qualitative evaluations of saliency maps, which often focus on "good" examples. Their proposed insertion/deletion metrics provide a more robust way to compare different saliency algorithms.
- Attention and activation mapping are similar: The study observes that attention-based and activation mapping saliency methods produce nearly identical results, suggesting a potential underlying connection.
- Self-similarity reveals class-specific features: Averaging self-similarity saliency maps across a class can highlight features that are consistently important for that class, offering a form of global explanation.
Key Results
- Image Retrieval Performance: The paper reports image retrieval results using mAP and P@K metrics. While not the primary focus, they show that their deep metric learning framework achieves reasonable performance on both datasets, with slightly better results on the COVID-19 dataset, likely due to the use of a pre-trained model and larger dataset size.
- Saliency Map Evaluation:
- Qualitative: Saliency maps generally focused on relevant regions (e.g., lung fields in X-rays) for correct retrievals. For incorrect retrievals, saliency often highlighted non-target areas. For skin lesions, saliency consistently focused on the lesion itself.
- Quantitative: On the COVID-19 dataset, occlusion-based saliency yielded the lowest deletion scores. For ISIC 2017, occlusion performed best on insertion, while attention and activation mapping excelled at deletion. Randomly initialized models showed significantly worse insertion and higher deletion scores, validating the metrics.
- Differential Saliency: Visualizations showed that differential saliency could highlight subtle differences between conditions, such as brighter lung regions for pneumonia/COVID-19 or specific contrast patterns in skin lesions.
Limitations and Future Work
- User Studies: The authors acknowledge the need for user studies with domain experts (radiologists, dermatologists) to confirm if X-MIR actually improves their performance on retrieval tasks.
- Integration with other explanations: Future work could explore combining saliency maps with other explanation modalities, such as textual explanations or medical report generation.
- Broader Applicability: The proposed methods are general and could be applied to other medical imaging problems (e.g., anomaly detection) and domains beyond medicine (e.g., face verification, person re-identification).
- Investigating Attention/Activation Similarity: Further research is needed to understand the observed similarity between attention-based and activation mapping saliency methods.
- Validation of Differential Saliency: The differential saliency findings are preliminary and require further investigation and validation.
Summary updated