Abstract
Deep neural networks have achieved tremendous success in image recognition, classification and object detection. However, deep learning is often criticised for its lack of transparency and general inability to rationalize its predictions. The issue of poor model interpretability becomes critical in medical applications, as a model that is not understood and trusted by physicians is unlikely to be used in daily clinical practice. In this work, we develop a novel multi-task deep learning framework for simultaneous histopathology image classification and retrieval, leveraging on the classic concept of k-nearest neighbors to improve model interpretability. For a test image, we retrieve the most similar images from our training databases. These retrieved nearest neighbours can be used to classify the test image with a confidence score, and provide a human-interpretable explanation of our classification. Our original framework can be built on top of any existing classification network (and therefore benefit from pretrained models), by (i) adding a triplet loss function with a novel triplet sampling strategy to compare distances between samples and (ii) a Cauchy hashing loss function to accelerate neighbour searching. We evaluate our method on colorectal cancer histology slides, and show that the confidence estimates are strongly correlated with model performance. The explanations provided by nearest neighbors are intuitive and useful for expert evaluation by giving insights into understanding possible model failures, and can support clinical decision making by comparing archived images and patient records with the actual case.
Introduction
Deep neural networks are highly effective at classifying medical images but are notoriously opaque "black boxes." In pathology, where diagnostic decisions are life-altering, physicians require a clear rationale to trust and adopt AI tools in clinical practice. Standard classification models often lack transparency and can be brittle when faced with data variations.
This paper introduces a multi-task deep learning framework that addresses these challenges by combining histopathological image classification with image retrieval. By employing a $k$-nearest neighbor ($k$-NN) approach, the model does not merely output a diagnosis; it presents the pathologist with the most similar training images used to reach that conclusion. This provides a human-interpretable "why" behind every prediction, fostering trust and enabling expert verification.
Methodology
The proposed framework is designed to be modular, allowing it to be integrated with various CNN backbones, such as ResNet. It optimizes three distinct loss functions simultaneously to achieve robust classification and efficient retrieval:
- Triplet Loss with Batch-Hard Sampling: This loss forces the network to map images into an embedding space where samples of the same tissue class are clustered together while different classes are pushed apart. An improved sampling strategy selects $k$ hard-negative samples, enhancing robustness against outliers.
- Cauchy Hashing Loss: To enable near-instantaneous searching through large databases, the model learns to represent each image as a short binary hash code. This allows for rapid searching in Hamming space rather than computationally expensive floating-point operations.
- Cross-Entropy Loss: This standard loss ensures the model maintains high diagnostic accuracy for the primary classification task.
The system employs a hierarchical retrieval strategy: a "coarse" search using binary hash codes identifies a candidate pool, followed by a "fine" search using $L2$ distance on high-dimensional feature vectors to select the top 10 most similar images.
Experimental Setup
The framework was evaluated using two primary datasets focused on colorectal cancer (CRC) histology:
- NCT-CRC-HE-100K: A large-scale training set comprising 100,000 HE-stained image patches across nine tissue classes (e.g., adipose, stroma, epithelium).
- CRC-VAL-HE-7K: An independent validation set of 7,180 patches from a different patient cohort. Utilizing an independent dataset is critical for assessing the model's ability to generalize across different laboratory environments and scanners.
Results and Discussion
The multi-task approach yielded significant improvements in both performance and interpretability:
- Superior Generalization: On the independent test set, the Multi-task ResNet18 achieved 95.0% accuracy, outperforming both the standard ResNet18 (94.4%) and the larger ResNet50 (93.6%). This suggests that the auxiliary retrieval task acts as a regularizer, helping the model learn more generalizable features.
- High-Precision Retrieval: Over 83% of query images (6,000+ out of 7,180) achieved "perfect" retrieval, where all 10 neighbors belonged to the correct class. This represents a 30% improvement over baseline classification-only networks.
- Interpretability and Error Analysis: By retrieving nearest neighbors, the AI mimics human reasoning by analogy. If the model misclassifies a patch, the retrieved neighbors often reveal the source of confusion—such as similar color profiles between "stroma" and "smooth muscle"—making the error immediately apparent to a pathologist.
Furthermore, the framework provides an inherent "confidence meter." The consistency of the retrieved neighbors serves as a proxy for model certainty; a mix of tissue classes in the top 10 neighbors flags the case for human review.
Conclusion
The integration of image retrieval into deep learning classification provides a powerful tool for interpretable medical AI. While challenges such as "domain shift" between datasets remain, this approach offers a path toward more transparent and trustworthy clinical decision support systems. Future work will focus on scaling this retrieval-based explanation to Whole Slide Images (WSI), providing pathologists with broader architectural context for complex cases.
Summary updated