Abstract
This paper introduces a Multiple Instance Learning (MIL) framework that forces deep learning models to base medical diagnoses on only a small subset of "key patches" within an image. By mimicking the way clinicians focus on specific localized findings, the researchers achieved diagnostic performance comparable to standard models while significantly improving robustness to dataset bias and providing clearer, more explainable results.
Introduction
Deep learning (DL) has shown incredible promise in medical imaging, but it suffers from a "shortcut" problem. Standard models often achieve high accuracy by latching onto spurious correlations—irrelevant features like skin tone, lighting, or even medical equipment artifacts—rather than actual clinical indicators. This leads to "dataset bias," where a model performs perfectly on the data it was trained on but fails when applied to different patient demographics or hospital settings.
Furthermore, most DL models are "black boxes." While a clinician uses specific criteria (like the 7-point checklist for skin lesions), a standard Convolutional Neural Network (CNN) or Vision Transformer (ViT) processes the entire image and aggregates information in a way that is difficult for humans to audit. The authors aim to bridge this gap by creating a model that is both robust to bias and inherently explainable.
Methods
The authors propose a framework that integrates Multiple Instance Learning (MIL) on top of existing feature extraction architectures. The process works in three main stages:
- Patch Encoder: The input image is processed by a backbone (either a CNN like EfficientNetB3 or a ViT like DEiT-S). Instead of pooling the entire image into one vector immediately, the model treats the image as a "bag" of patches (instances).
- MIL Block: The model evaluates these patches to identify the most discriminative ones. The authors explored two approaches: Instance-level (predicting a class for each patch) and Embedding-level (aggregating feature vectors first).
- Top-$k$ Pooling: This is the "secret sauce." Instead of looking at every patch, the model uses a permutation-invariant pooling operator (specifically top-$k$ average pooling) to select only the most relevant regions (e.g., the top 25% of patches).
By limiting the information the classifier can use, the model is forced to ignore the "background" and focus on the actual pathology.
Datasets
The framework was evaluated on two distinct medical domains:
- Skin Cancer (Dermoscopy): Primary training used ISIC 2019 (over 25,000 images), with HIBA, PH², and Derm7pt used as out-of-domain test sets to evaluate generalization.
- Breast Cancer (Mammography): The DDSM dataset was used for binary classification (findings vs. no findings).
Key Insights and Results
One of the most striking findings is that less is more: discarding a large portion of the image data (up to 75% or more) does not compromise diagnostic performance and often improves it. By forcing the model to pick "key patches," it becomes much harder for the model to "cheat" using background artifacts.
- Generalization Boost: On the HIBA dataset (unseen during training), the MIL-based CNN outperformed the standard CNN baseline by 3.6% in Balanced Accuracy (BA).
- Performance Stability: In the ISIC 2019 multi-class task, the MIL-DEiT-S model achieved a BA of 82.6%, competitive with full-image models while using significantly less information.
- Qualitative Clarity: Heatmaps generated by the MIL framework (using Grad-CAM) were much more localized and detailed compared to the "blurry" heatmaps of standard models.
As shown in Figure 4, the framework accurately highlights the borders and internal structures of lesions, making the AI a more trustworthy assistant by aligning with clinical observation patterns.
Limitations and Future Work
The value of $k$ (how many patches to keep) currently needs to be defined empirically. Finding a way to make the model learn the optimal $k$ automatically would be a significant step forward. Additionally, the current MIL approach sometimes processes patches independently, potentially losing spatial context. Future work should involve validating these identified regions against specific medical concepts (like "pigment networks") with the help of medical experts.
Summary updated