Deep Interpretable Classification and Weakly-Supervised Segmentation of Histology Images via Max-Min Uncertainty

Soufiane Belharbi, Jérôme Rony, Jose Dolz, Ismail Ben Ayed, Luke McCaffrey, Eric Granger

TL;DR

This paper introduces a "Max-Min Uncertainty" framework for histology image analysis, allowing models to segment tumors and glands using only image-level labels (e.g., "cancer" vs. "healthy"). By forcing the model to be highly confident about foreground regions and highly uncertain about background regions, the authors significantly reduce false positives and improve interpretability in complex medical scans.

What Problem Does This Paper Address?

In medical imaging, particularly histology (the study of tissue under a microscope), obtaining pixel-perfect annotations is incredibly expensive and time-consuming. Weakly-supervised learning (WSL) attempts to solve this by training models using only global image labels.

However, standard WSL methods (like Class Activation Maps) often struggle with histology images because the "foreground" (cancerous cells) and "background" (healthy tissue) often share similar visual features. This leads to "over-segmentation," where the model incorrectly identifies large swaths of healthy tissue as diseased. Most existing methods focus only on finding discriminative features; they lack a formal mechanism to model what the background should look like to the classifier.

Methods

The authors propose a dual-branch architecture consisting of a Localizer ($M$) and a Classifier ($C$). The core innovation is the Max-Min Uncertainty objective:

  1. Foreground Localization ($X^+$): The localizer identifies a region that the classifier should be very certain about. The model minimizes standard cross-entropy here, pushing the prediction toward the ground-truth label.
  2. Background Localization ($X^-$): This is the "Max" part of the uncertainty. The localizer identifies a region where the classifier should be maximally uncertain. The authors use Kullback-Leibler (KL) divergence to force the classifier’s output for the background to match a uniform distribution (i.e., the model has no idea which class it belongs to).
  3. Absolute Size Constraints (ASC): To prevent the model from taking the easy way out (like labeling the entire image as foreground), they implement log-barrier regularization terms. These terms penalize extremely imbalanced segmentations, encouraging the model to find the "largest reasonable" regions for both foreground and background.

The authors tested two variants of the KL loss: Explicit Entropy Maximization (EEM) and a Surrogate for Explicit Entropy Maximization (SEM), which differ in their gradient dynamics but share the same goal of promoting high uncertainty in non-discriminative regions.

Datasets

The framework was evaluated on two public histology benchmarks:

  • GlaS (Gland Segmentation) dataset: 165 images of colon tissue sections used for cancer grading.
  • Camelyon16 patch-based benchmark: A large-scale dataset (~48,000 patches) derived from whole-slide images of breast cancer metastases in lymph nodes.

Key Insights and Ideas

  • Uncertainty as a Map: Instead of just looking for "features," the model uses the absence of evidence as a defining characteristic of the background. If a region doesn't help the classifier make a decision, it belongs in the background mask.
  • The Trivial Solution Trap: In WSL, models often converge to a "trivial solution" where they select the whole image as the object of interest. The use of log-barrier functions (borrowed from convex optimization) provides a "soft" way to constrain region sizes without requiring the user to know the exact size of the tumor beforehand.
  • Interpretability: By explicitly masking the background, the model provides a clear visual explanation of why it classified an image a certain way, making it a "white-box" tool that pathologists can more easily trust.

Key Results

The proposed method showed substantial improvements over state-of-the-art WSL methods like Deep MIL and Wildcat:

  • GlaS Dataset: The model achieved a 72.11% F1+ score (foreground segmentation), a 3.5% boost over the next best method. More impressively, the F1- score (background accuracy) jumped from ~41% in previous methods to ~69%, indicating a massive reduction in false positives.
  • Camelyon16: The model maintained competitive classification accuracy while providing much cleaner segmentation masks, especially in "normal" patches where other models frequently hallucinated tumors.
  • Classification Accuracy: On the GlaS test set, the model achieved a 0% classification error, matching or beating all other weakly-supervised competitors.

Limitations and Future Work

  • Binary Focus: The current work focuses primarily on binary classification (e.g., benign vs. malignant). The authors note that extending this to multi-class scenarios is a logical next step.
  • Computational Overhead: The method requires two passes (one for the localizer and one for the classifier) during training, though inference remains fast (less than a second per image).
  • Prior Knowledge: While the size constraints are "soft," the model still benefits from hyper-parameters that roughly guide the expected scale of objects. Future work could explore ways to make these constraints even more autonomous by incorporating texture or shape priors.

Summary updated

Abstract

Weakly-supervised learning (WSL) has recently triggered substantial interest as it mitigates the lack of pixel-wise annotations. Given global image labels, WSL methods yield pixel-level predictions (segmentations), which enable to interpret class predictions. Despite their recent success, mostly with natural images, such methods can face important challenges when the foreground and background regions have similar visual cues, yielding high false-positive rates in segmentations, as is the case in challenging histology images. WSL training is commonly driven by standard classification losses, which implicitly maximize model confidence, and locate the discriminative regions linked to classification decisions. Therefore, they lack mechanisms for modeling explicitly non-discriminative regions and reducing false-positive rates. We propose novel regularization terms, which enable the model to seek both non-discriminative and discriminative regions, while discouraging unbalanced segmentations. We introduce high uncertainty as a criterion to localize non-discriminative regions that do not affect classifier decision, and describe it with original Kullback-Leibler (KL) divergence losses evaluating the deviation of posterior predictions from the uniform distribution. Our KL terms encourage high uncertainty of the model when the latter inputs the latent non-discriminative regions. Our loss integrates: (i) a cross-entropy seeking a foreground, where model confidence about class prediction is high; (ii) a KL regularizer seeking a background, where model uncertainty is high; and (iii) log-barrier terms discouraging unbalanced segmentations. Comprehensive experiments and ablation studies over the public GlaS colon cancer data and a Camelyon16 patch-based benchmark for breast cancer show substantial improvements over state-of-the-art WSL methods, and confirm the effect of our new regularizers (our code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/sbelharbi/deep-wsl-histo-min-max-uncertainty</uri> ).
BibTeX
@article{belharbi2022deep,
  author = {Soufiane Belharbi and Jérôme Rony and Jose Dolz and Ismail Ben Ayed and Luke McCaffrey and Eric Granger},
  title = {{Deep Interpretable Classification and Weakly-Supervised Segmentation of Histology Images via Max-Min Uncertainty}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2022},
  doi = {10.1109/tmi.2021.3123461},
}

Loading related content...

Figure