Using Variational Autoencoders for Out of Distribution Detection in Histological Multiple Instance Learning

Francisco Javier Sáez-Maldonado, Luz García, Lee A. D. Cooper, Jeffery A. Goldstein, Rafael Molina, Aggelos K. KatsaggelosIEEE Access 2025 · DOI

TL;DR

This paper introduces VAEABMIL, a novel deep learning model for histological image analysis that can both classify tissue types and detect "out-of-distribution" (OOD) samples – images that don't conform to the training data. This is crucial for real-world medical applications where unexpected variations can occur, and their method demonstrates strong performance in both classification and OOD detection.

What Problem Does This Paper Address?

In digital pathology, classifying whole slide images (WSIs) often uses Multiple Instance Learning (MIL), which reduces annotation effort by only requiring labels at the WSI level. However, a critical challenge for deploying these systems in real-world scenarios is their inability to identify "out-of-distribution" (OOD) samples. These are images that deviate from the training data due to unseen tissues, artifacts, or other variations. Current MIL models typically fail when encountering such data, potentially leading to incorrect diagnoses. This paper addresses this gap by developing a MIL model that is explicitly designed to detect OOD samples, thereby increasing the reliability and safety of computer-assisted diagnosis systems.

Methods

The core of the proposed method, VAEABMIL, is a novel probabilistic deep MIL model that combines a Variational Autoencoder (VAE) with an Attention-Based MIL (ABMIL) architecture.

  • Variational Autoencoder (VAE): The VAE is used to model the underlying data distribution of the instances (image patches) within a WSI. This allows the model to learn a probabilistic latent representation for each instance, which is key for OOD detection.
  • Attention-Based MIL (ABMIL): The ABMIL component is used for the classification task. It leverages the attention mechanism to weigh the importance of each instance's latent representation for classifying the entire bag (WSI).
  • OOD Detection: The probabilistic latent representations from the VAE are used to compute an instance-level marginal likelihood. By aggregating these likelihoods across all instances in a WSI, a bag-level OOD score (LOGPX) is derived. A lower marginal likelihood indicates a higher probability of the sample being OOD.
  • Deterministic Version (DAEABMIL): The authors also propose a deterministic version, DAEABMIL, which uses the reconstruction error of the VAE as an OOD score. This version is simpler to optimize and offers faster inference.

Datasets

The authors used four datasets for their experiments:

  • Camelyon16 (CAM16): Used for training and testing in-distribution (IND) data for breast cancer metastasis detection.
  • Panda (PANDA): Used for training and testing IND data for prostate cancer classification.
  • BCELL: Used as an OOD dataset for CAM16. Both datasets contain lymph node tissue.
  • ARTIF: Used as an OOD dataset for PANDA. Both datasets contain prostate tissue and ARTIF includes various artifacts.

Key Insights and Ideas

  • Modeling Data Distribution is Key for OOD Detection: The paper strongly suggests that explicitly modeling the data distribution (as done by the VAE) is crucial for effective OOD detection, unlike traditional methods that rely on model-agnostic scores.
  • Probabilistic Latent Space Benefits OOD Detection: The continuous and probabilistic nature of the VAE's latent space in VAEABMIL appears to be more robust for OOD detection compared to the deterministic latent space in DAEABMIL, especially in Far OOD scenarios.
  • Foundation Models Enhance Performance: Using pre-trained foundation models (like UNI) as feature extractors significantly improves both classification and OOD detection performance across all models, highlighting their importance for downstream tasks.
  • Joint Optimization is Challenging but Rewarding: While jointly optimizing for classification and OOD detection makes the models harder to train, it leads to superior OOD detection capabilities without significantly sacrificing classification performance.

Key Results

  • Classification Performance: VAEABMIL and DAEABMIL achieve classification AUC scores comparable to state-of-the-art MIL models on both CAM16 and PANDA datasets, often exceeding 0.95 AUC, especially when using the UNI feature extractor.
  • OOD Detection Performance:
    • Far OOD: VAEABMIL consistently outperforms other models in Far OOD detection scenarios (e.g., CAM16 vs. PANDA), achieving AUCs of 100% in some cases. DAEABMIL also performs well, though sometimes slightly behind VAEABMIL.
  • Near OOD: In more challenging Near OOD scenarios (e.g., CAM16 vs. BCELL, PANDA vs. ARTIF), VAEABMIL and DAEABMIL excel, achieving near-perfect AUCs. This is a significant advantage over other models, which struggle to differentiate between similar tissue types with subtle differences. For instance, in the (CAM16, BCELL) experiment, VAEABMIL and DAEABMIL achieve AUCs of 100% and 97% respectively.
  • Artifact Detection: The models demonstrate the ability to identify specific regions of artifacts within WSIs, as visualized in Figure 13.

Limitations and Future Work

  • Optimization Difficulty: The primary limitation acknowledged is that VAEABMIL and DAEABMIL are harder to optimize than other deep MIL models due to the multi-objective loss function (classification and OOD detection). This can lead to slower convergence and, in some cases, a slight performance decrease for DAEABMIL.
  • Future Work: The authors suggest extending their approach by combining VAEs with more complex MIL methods (like TRANSMIL or DTFDMIL) and exploring other generative models for data distribution learning to further enhance OOD detection performance.

Summary updated

BibTeX
@article{saezmaldonado2025using,
  author = {Francisco Javier Sáez-Maldonado and Luz García and Lee A. D. Cooper and Jeffery A. Goldstein and Rafael Molina and Aggelos K. Katsaggelos},
  title = {{Using Variational Autoencoders for Out of Distribution Detection in Histological Multiple Instance Learning}},
  journal = {IEEE Access},
  year = {2025},
  doi = {10.1109/access.2025.3593420},
}
Figure