INSIGHT: Explainable Weakly-Supervised Medical Image Analysis

Wenbo Zhang, Junyu Chen, Christopher KananarXiv 2024 · DOI · arXiv

Introduction

Medical imaging modalities such as Whole-Slide Images (WSIs) and CT scans present a significant computational challenge due to their massive scale—often containing billions of pixels or hundreds of slices. To process these, researchers typically employ "divide-and-conquer" pipelines that extract small patch embeddings for global prediction. However, these aggregators often suffer from a loss of spatial context and rely on noisy post-hoc visualization tools like Grad-CAM, which can miss clinically vital details and produce blurry results.

The INSIGHT Framework

The INSIGHT (Integrated Network for Segmentation and Interpretation with Generalized Heatmap Transmission) architecture addresses these issues by building interpretability directly into the model. Unlike traditional methods that require expensive pixel-level annotations for segmentation, INSIGHT operates under a weakly-supervised paradigm, using only image-level labels to generate high-quality segmentation heatmaps.

Comparison of annotation costs showing how INSIGHT provides interpretable heatmaps "for free" compared to expensive pixel-level labeling.

Key architectural innovations include:

  • Dual-Module Architecture: A Detection Module captures fine-grained textures, while a Context Module uses larger kernels to capture broader spatial relationships and suppress false positives in irrelevant regions.
  • Heatmap Fusion & Dynamic Thresholding: Module outputs are fused into an internal heatmap, which is then sharpened using Otsu’s algorithm to filter background noise.
  • SmoothMax Pooling: By considering all relevant activations rather than just the single most positive patch, SmoothMax allows the model to learn more accurate lesion boundaries and account for the entire area of a lesion.

Key Results and Insights

INSIGHT was evaluated on three major benchmarks: MosMed (CT), CAMELYON16 (WSI), and BRACS (WSI). The framework achieved state-of-the-art performance in both classification and segmentation:

  • Classification: Reached a 96.2% AUC on the MosMed CT dataset, nearly 5% higher than the best baseline.
  • Segmentation: Achieved a 74.6% Dice score on CAMELYON16, outperforming competing models by 6.9%.
Qualitative results of INSIGHT on a Whole-Slide Image (WSI), showing precise lesion localization at both global and zoomed-in scales.

A critical finding is that forcing the model to generate heatmaps during the forward pass acts as a regularizer, improving classification accuracy by preventing the model from focusing on background artifacts. Furthermore, INSIGHT demonstrates a superior ability to detect small lesions that are often "blurred away" by standard Multiple Instance Learning (MIL) methods.

Conclusion

INSIGHT represents a shift toward "explainable-by-design" architectures in medical AI. By maintaining spatial resolution and accounting for the entire lesion area, it provides high-fidelity clinical insights without the need for dense annotations. Future work will explore its application to 3D volumetric data and integration with emerging foundation models.

Summary updated

Abstract

Due to their large sizes, volumetric scans and whole-slide pathology images (WSIs) are often processed by extracting embeddings from local regions and then an aggregator makes predictions from this set. However, current methods require post-hoc visualization techniques (e.g., Grad-CAM) and often fail to localize small yet clinically crucial details. To address these limitations, we introduce INSIGHT, a novel weakly-supervised aggregator that integrates heatmap generation as an inductive bias. Starting from pre-trained feature maps, INSIGHT employs a detection module with small convolutional kernels to capture fine details and a context module with a broader receptive field to suppress local false positives. The resulting internal heatmap highlights diagnostically relevant regions. On CT and WSI benchmarks, INSIGHT achieves state-of-the-art classification results and high weakly-labeled semantic segmentation performance. Project website and code are available at: https://zhangdylan83.github.io/ewsmia/
BibTeX
@misc{zhang2024insight,
  author = {Wenbo Zhang and Junyu Chen and Christopher Kanan},
  title = {{INSIGHT: Explainable Weakly-Supervised Medical Image Analysis}},
  year = {2024},
  eprint = {2412.02012},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.02012},
}
Figure