Key Patches Are All You Need: A Multiple Instance Learning Framework For Robust Medical Diagnosis

D. J. Araújo, Maria Rita Verdelho, Alceu Bissoto, Jacinto C. Nascimento, Carlos Santiago, Catarina BarataWorkshops @ CVPR 2024 · DOI · arXiv

Abstract

This paper introduces a Multiple Instance Learning (MIL) framework that forces deep learning models to base medical diagnoses on only a small subset of "key patches" within an image. By mimicking the way clinicians focus on specific localized findings, the researchers achieved diagnostic performance comparable to standard models while significantly improving robustness to dataset bias and providing clearer, more explainable results.

Introduction

Deep learning (DL) has shown incredible promise in medical imaging, but it suffers from a "shortcut" problem. Standard models often achieve high accuracy by latching onto spurious correlations—irrelevant features like skin tone, lighting, or even medical equipment artifacts—rather than actual clinical indicators. This leads to "dataset bias," where a model performs perfectly on the data it was trained on but fails when applied to different patient demographics or hospital settings.

Furthermore, most DL models are "black boxes." While a clinician uses specific criteria (like the 7-point checklist for skin lesions), a standard Convolutional Neural Network (CNN) or Vision Transformer (ViT) processes the entire image and aggregates information in a way that is difficult for humans to audit. The authors aim to bridge this gap by creating a model that is both robust to bias and inherently explainable.

Methods

The authors propose a framework that integrates Multiple Instance Learning (MIL) on top of existing feature extraction architectures. The process works in three main stages:

  1. Patch Encoder: The input image is processed by a backbone (either a CNN like EfficientNetB3 or a ViT like DEiT-S). Instead of pooling the entire image into one vector immediately, the model treats the image as a "bag" of patches (instances).
  2. MIL Block: The model evaluates these patches to identify the most discriminative ones. The authors explored two approaches: Instance-level (predicting a class for each patch) and Embedding-level (aggregating feature vectors first).
  3. Top-$k$ Pooling: This is the "secret sauce." Instead of looking at every patch, the model uses a permutation-invariant pooling operator (specifically top-$k$ average pooling) to select only the most relevant regions (e.g., the top 25% of patches).
The MIL framework architecture, showing the transition from input image to patch encoding and subsequent instance-level or embedding-level learning.

By limiting the information the classifier can use, the model is forced to ignore the "background" and focus on the actual pathology.

Datasets

The framework was evaluated on two distinct medical domains:

  • Skin Cancer (Dermoscopy): Primary training used ISIC 2019 (over 25,000 images), with HIBA, PH², and Derm7pt used as out-of-domain test sets to evaluate generalization.
  • Breast Cancer (Mammography): The DDSM dataset was used for binary classification (findings vs. no findings).

Key Insights and Results

One of the most striking findings is that less is more: discarding a large portion of the image data (up to 75% or more) does not compromise diagnostic performance and often improves it. By forcing the model to pick "key patches," it becomes much harder for the model to "cheat" using background artifacts.

  • Generalization Boost: On the HIBA dataset (unseen during training), the MIL-based CNN outperformed the standard CNN baseline by 3.6% in Balanced Accuracy (BA).
  • Performance Stability: In the ISIC 2019 multi-class task, the MIL-DEiT-S model achieved a BA of 82.6%, competitive with full-image models while using significantly less information.
  • Qualitative Clarity: Heatmaps generated by the MIL framework (using Grad-CAM) were much more localized and detailed compared to the "blurry" heatmaps of standard models.
Qualitative comparison of heatmaps and key patches for instance-level MIL models using max pool and top-k average pool.

As shown in Figure 4, the framework accurately highlights the borders and internal structures of lesions, making the AI a more trustworthy assistant by aligning with clinical observation patterns.

Limitations and Future Work

The value of $k$ (how many patches to keep) currently needs to be defined empirically. Finding a way to make the model learn the optimal $k$ automatically would be a significant step forward. Additionally, the current MIL approach sometimes processes patches independently, potentially losing spatial context. Future work should involve validating these identified regions against specific medical concepts (like "pigment networks") with the help of medical experts.

Summary updated

BibTeX
@inproceedings{araujo2024key,
  author = {D. J. Araújo and Maria Rita Verdelho and Alceu Bissoto and Jacinto C. Nascimento and Carlos Santiago and Catarina Barata},
  title = {{Key Patches Are All You Need: A Multiple Instance Learning Framework For Robust Medical Diagnosis}},
  booktitle = {Workshops @ CVPR},
  year = {2024},
  doi = {10.1109/cvprw63382.2024.00532},
}
Figure