Explaining Digital Pathology Models via Clustering Activations

Adam Bajger, Jan Obdrzálek, Vojtech Kur, Rudolf Nenutil, Petr Holub, Vít Musil, Tomás BrázdilarXiv 2025 · DOI · arXiv

Abstract

Deep learning models demonstrate remarkable efficacy in detecting malignancy within Whole Slide Images (WSIs), yet their "black box" nature limits clinical trust. We propose a novel explainability framework that utilizes Non-negative Matrix Factorization (NMF) to cluster internal activations of Convolutional Neural Networks (CNNs). Unlike traditional saliency maps that merely highlight regions of interest, our method generates global, semantically meaningful tissue segmentations. These clusters align with specific morphological features recognized by pathologists, bridging the gap between automated classification and human-interpretable diagnostics.

Introduction: The Interpretability Gap

While deep learning has revolutionized digital pathology, a significant "trust gap" persists. Pathologists require more than binary classifications; they need to understand the morphological reasoning behind a model's decision. Existing eXplainable AI (XAI) techniques, such as GradCAM, produce one-dimensional saliency maps. While these indicate where a model is looking, they fail to describe what morphological features—such as gland structure or cell density—are driving the prediction.

Traditional GradCAM saliency maps highlight regions of importance but lack morphological specificity.

Methodology

The proposed approach interprets a VGG-16 model trained on prostate cancer detection through three primary steps:

  1. Activation Extraction: Feature vectors are extracted from the final convolutional layer of the trained model.
  2. NMF Clustering: Utilizing the non-negative nature of ReLU activations, the authors use Non-negative Matrix Factorization (NMF) to decompose these vectors into $K$ distinct classes. This allows for the visualization of "class intensity" across spatial locations.
  3. Global Visualization: The resulting clusters are overlaid on the original H&E stained tissue, creating a semantic map based purely on the model's learned representations.

Results and Discussion

The study reveals that CNNs naturally learn to distinguish between various tissue types (e.g., stroma vs. epithelium) even when only trained for binary cancer classification.

Pathological Alignment

A resident pathologist confirmed that the generated clusters correspond to distinct histological structures. For instance, specific clusters were identified as "fascicular structures with sparse elongated nuclei" or "chains of densely packed nuclei," providing a level of granularity absent in standard heatmaps.

The proposed clustering method provides a multi-colored semantic segmentation of tissue types based on model activations.

Quantitative Validation

To ensure these clusters captured the model's decision-making logic, the researchers trained a linear proxy model using only cluster weights. This proxy achieved an AUC of 0.998, indicating that the NMF clusters retain nearly all the predictive information used by the original "black box" CNN. The analysis identified specific clusters (Classes 2, 4, and 5) that strongly correlate with cancerous regions, while others represent benign or background tissue.

Conclusion

By moving from simple heatmaps to structured clustering, this method provides a transparent view into the internal logic of pathology models. This approach not only identifies where a model is looking but provides a semantically grounded explanation of what it sees. Future work will focus on scaling this approach to modern transformer architectures and evaluating its impact on clinical workflows.

Summary updated

Abstract

We present a clustering-based explainability technique for digital pathology models based on convolutional neural networks. Unlike commonly used methods based on saliency maps, such as occlusion, GradCAM, or relevance propagation, which highlight regions that contribute the most to the prediction for a single slide, our method shows the global behaviour of the model under consideration, while also providing more fine-grained information. The result clusters can be visualised not only to understand the model, but also to increase confidence in its operation, leading to faster adoption in clinical practice. We also evaluate the performance of our technique on an existing model for detecting prostate cancer, demonstrating its usefulness.
BibTeX
@misc{bajger2025explaining,
  author = {Adam Bajger and Jan Obdrzálek and Vojtech Kur and Rudolf Nenutil and Petr Holub and Vít Musil and Tomás Brázdil},
  title = {{Explaining Digital Pathology Models via Clustering Activations}},
  year = {2025},
  eprint = {2511.14558},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.14558},
}
Figure