High- and Low-level image component decomposition using VAEs for improved reconstruction and anomaly detection

David Zimmerer, Jens Petersen, Klaus H. Maier-HeinarXiv 2019 · arXiv

Introduction

Variational Auto-Encoders (VAEs) are a cornerstone of unsupervised learning in medical imaging, particularly for tasks such as feature extraction and anomaly detection. However, standard VAEs often produce blurry, over-smoothed reconstructions due to their tendency to prioritize low-level pixel statistics over high-level semantic features. While hierarchical architectures like the Ladder VAE attempt to address this by learning multi-scale features, they frequently suffer from "posterior collapse," where the model ignores the hierarchy and reverts to a single-level representation. This paper introduces the primary components conditional hierarchical VAE (pchVAE), which explicitly decomposes images into high-level (coarse) and low-level (fine) components to achieve sharper reconstructions and more robust anomaly detection.

Methodology

The pchVAE architecture is designed to prevent posterior collapse by enforcing a structural partition between global structure and textural detail.

The pchVAE architecture explicitly separates high-level and low-level latent spaces to prevent posterior collapse and ensure sharp reconstructions.

Explicit Partitioning and Information Penalty

Inspired by the VQVAE-2 and PCA, the model utilizes a dedicated low-level reconstruction branch. To ensure the latent spaces remain distinct, the authors introduce a novel optimization objective featuring a mutual information penalty. This loss term minimizes the shared information between the high-level and low-level components, forcing the high-level latent space to encode global organ structure while the low-level space captures residual fine-grained textures.

Two-Pass Reconstruction

The final image is generated through a two-pass process. A high-level decoder first reconstructs the coarse structure, which then conditions a low-level decoder. The final output is an addition of these two components, ensuring that both semantic and textural information are preserved without the "hallucinations" common in Generative Adversarial Networks (GANs).

Experimental Setup

The models were trained on 800 healthy brain MRI slices from the Human Connectome Project (HCP). Evaluation was conducted on:

  1. Reconstruction: 200 held-back HCP patients.
  2. Synthetic Anomaly Detection: HCP slices with randomly rendered natural objects.
  3. Real Medical Anomaly Detection: The BraTS2017 dataset, featuring expert-annotated brain tumors as out-of-distribution (OoD) samples.

Results

The pchVAE was benchmarked against several baselines, including VQVAE-2 and variants focusing solely on high or low levels.

Qualitative comparison of brain MRI reconstructions. The pchVAE successfully captures both the global structure (high) and fine details (low) compared to baseline models.

Quantitative Performance

The pchVAE demonstrated superior performance in anomaly detection, achieving an AUROC of 0.7277 and an Average Precision (AP) of 0.5321, outperforming VQVAE-2 (AUROC 0.6965). In terms of reconstruction, the pchVAE achieved a Mean Squared Error (MSE) of 0.03224. While a purely texture-focused "Low VAE" achieved lower MSE (0.01169), it failed at anomaly detection, suggesting that low MSE alone does not indicate a model's understanding of the underlying semantic distribution.

Conclusion

By explicitly decomposing image components, the pchVAE overcomes the blurriness and posterior collapse issues inherent in traditional VAEs. This approach provides a safer alternative to GANs for medical applications, as it achieves sharpness through hierarchical decomposition rather than adversarial "dreaming." Future work will explore the application of this decomposition for precise anomaly localization and more complex 3D medical datasets.

Summary updated

Abstract

Variational Auto-Encoders have often been used for unsupervised pretraining, feature extraction and out-of-distribution and anomaly detection in the medical field. However, VAEs often lack the ability to produce sharp images and learn high-level features. We propose to alleviate these issues by adding a new branch to conditional hierarchical VAEs. This enforces a division between higher-level and lower-level features. Despite the additional computational overhead compared to a normal VAE it results in sharper and better reconstructions and can capture the data distribution similarly well (indicated by a similar or slightly better OoD detection performance).
BibTeX
@misc{zimmerer2019high,
  author = {David Zimmerer and Jens Petersen and Klaus H. Maier-Hein},
  title = {{High- and Low-level image component decomposition using VAEs for improved reconstruction and anomaly detection}},
  year = {2019},
  eprint = {1911.12161},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.12161},
}
Figure