Rethinking Autoencoders for Medical Anomaly Detection from A Theoretical Perspective

Yu Cai, Hao Chen, Kwang-Ting Cheng

Abstract

Autoencoders (AEs) are widely used for medical anomaly detection through reconstruction error analysis. However, their tendency to generalize too well—reconstructing anomalies they were never trained on—remains a significant challenge. This paper provides a theoretical foundation for this phenomenon, proving that "identical shortcuts" can be avoided by constraining the latent dimension. We introduce the Entropy Matching Principle, suggesting that optimal performance is achieved when the latent space entropy matches that of the normal data. Our findings, validated across four medical datasets, show that simple AEs with optimized bottlenecks often outperform complex, specialized architectures.

Introduction

In medical imaging, anomalies such as tumors or lesions are rare and highly variable, making supervised classification difficult. Consequently, reconstruction-based Anomaly Detection (AD) has become a standard approach. The core idea is to train an Autoencoder (AE) exclusively on "normal" healthy images. During inference, the model is expected to reconstruct healthy tissue accurately while failing to reconstruct unseen anomalies. The resulting reconstruction error serves as an anomaly map.

Autoencoder architecture for reconstruction-based medical anomaly detection.

A persistent issue in this field is the "identical shortcut," where deep neural networks learn to pass pixels through the bottleneck without extracting meaningful features, leading to perfect reconstruction of both normal and abnormal data. While previous research relied on heuristic "tricks" like memory banks or attention mechanisms to mitigate this, a rigorous theoretical explanation has been lacking.

Theoretical Framework

The authors utilize Information Theory to address these failures through two primary propositions:

  1. The Death of the "Identical Shortcut" (Proposition 1): Mathematically, if the latent dimension ($d$) is less than half the dimension of the preceding feature layer ($D/2$), the network is physically incapable of learning an identical mapping. The shortcut is thus a direct consequence of excessive bottleneck capacity.
  2. The Entropy Matching Principle (Proposition 2): Using Mutual Information, the authors argue that an optimal AE for anomaly detection must have a latent space entropy $H(Z)$ that matches the entropy of normal data $H(X_n)$. If $H(Z)$ is too high, the model captures the additional information present in lesions, resulting in undesirable "perfect reconstruction" of anomalies.

By adjusting the latent dimension $d$, one can control this entropy and force the model to ignore abnormal patterns.

Experimental Results

The theory was validated across four datasets: RSNA and VinDr-CXR (Chest X-rays), and Brain Tumor and BraTS2021 (Brain MRIs). These datasets represent varying levels of informational complexity, with MRIs possessing significantly higher entropy than X-rays.

Performance metrics (AUC and AP) across various datasets and latent dimensions.

Key Insights

  • Optimal Bottleneck Size: For lower-entropy X-ray data, the best performance was achieved with a very small latent dimension ($d=4$). Increasing $d$ beyond this point allowed the model to begin "memorizing" abnormal textures, degrading detection performance.
  • Complexity Correlation: Higher-complexity MRI scans required larger latent dimensions (16 or 32) to capture the necessary healthy tissue information, confirming that the latent space must scale with the complexity of the normal data.
  • Superiority of Simple AEs: A theoretically-tuned simple AE consistently outperformed more complex models like Memory-augmented AEs (MemAE) and Variational Autoencoders (VAE). For instance, on the RSNA dataset, optimizing $d$ from 128 to 4 resulted in a 10% improvement in AUC.

Conclusion

This work demonstrates that the success of medical anomaly detection lies not in architectural complexity, but in the careful management of information flow through the network bottleneck. While the current selection of the optimal latent dimension $d$ requires manual tuning, future research will focus on developing "self-adaptive" Autoencoders capable of automatically quantifying training data entropy to set the bottleneck size dynamically.

Summary updated

BibTeX
@inproceedings{cai2024rethinking,
  author = {Yu Cai and Hao Chen and Kwang-Ting Cheng},
  title = {{Rethinking Autoencoders for Medical Anomaly Detection from A Theoretical Perspective}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2024},
  doi = {10.1007/978-3-031-72120-5\_51},
}
Figure