Technical outlier detection via convolutional variational autoencoder for the ADMANI breast mammogram dataset

Hui Li, Carlos A. Peña-Solórzano, Susan Wei, Davis J. McCarthyarXiv 2023 · DOI · arXiv

Abstract

Large-scale medical imaging datasets are essential for training robust artificial intelligence models, yet they are frequently contaminated with technical artifacts that can confound learning. This paper presents a hybrid unsupervised pipeline designed to detect technical outliers in the ADMANI (Annotated Digital Mammograms and Associated Non-Image) dataset. By combining Convolutional Variational Autoencoders (CVAE) with traditional morphological operations and pectoral muscle analysis, the researchers achieved a significant improvement in outlier detection, raising recall from 31% to 83%. The findings underscore the limitations of deep learning in isolating small, high-density artifacts and demonstrate the efficacy of hybridizing modern neural networks with classical computer vision techniques.

Introduction

In the development of AI for breast cancer screening, the principle of "garbage in, garbage out" remains a significant hurdle. High-quality training requires massive datasets, such as the ADMANI database, which contains over 4.4 million mammograms. However, real-world clinical data is rarely pristine. It often contains "technical outliers"—images compromised not by pathology, but by external factors such as implanted medical devices or radiography errors.

If left unaddressed, these outliers act as confounders; an AI might erroneously learn to associate a specific artifact, like a pacemaker lead, with a clinical diagnosis. Manually auditing millions of images is logistically impossible. This research explores an automated, unsupervised methodology to flag these problematic images, ensuring the integrity of the downstream training process.

Defining Technical Outliers

The study categorizes technical outliers into two primary groups:

  1. Implanted Devices: These include pacemakers, breast implants, and cardiac loop recorders. These objects introduce high-density signals that do not represent biological tissue.
  2. Radiography Errors: These encompass improper patient positioning, incorrect exposure settings, or "heterogeneous" pectoral muscle projections that obscure the underlying breast tissue.

Methodology

The authors proposed a multi-stage unsupervised pipeline that integrates deep learning with hand-crafted image processing rules.

1. Convolutional Variational Autoencoders (CVAE)

The core of the detection system is a CVAE trained to learn the latent representation of a "normal" mammogram. The hypothesis is that the model, having learned the distribution of standard images, will fail to accurately reconstruct an outlier, resulting in high reconstruction loss. The researchers compared a 5-layer VanillaCVAE with a deeper 18-layer ResNetCVAE, discovering that the simpler architecture was more effective for this specific outlier detection task.

To identify outliers, the team evaluated 15 different scoring metrics, including:

  • Generative Loss: Reconstruction loss, Kullback-Leibler Divergence (KLD), and Evidence Lower Bound (ELBO).
  • Latent Space Analysis: Applying algorithms such as Isolation Forest (IF), Local Outlier Factor (LOF), and One-Class SVM (OCSVM) to the compressed vectors generated by the CVAE.

2. Hand-Crafted Supplements

Initial testing revealed that the CVAE struggled to identify small, high-density objects. To compensate, the authors introduced two classical computer vision steps:

  • Morphological Erosion: This operation was used to isolate small, bright regions, making it highly effective at catching pacemakers and cardiac recorders.
  • Pectoral Muscle Analysis: Utilizing Canny edge detection and Hough transforms, the system identified images where the pectoral muscle was improperly imaged or overly complex, which often indicates poor positioning.

Experimental Results

The system was tested on a subset of 30,000 mammograms from the ADMANI dataset. A professional radiologist established the ground truth by identifying 136 true outliers (approximately 0.465% of the subset).

Performance was measured by Recall—the percentage of true outliers caught within the top X% of the most suspicious images identified by the system:

  • CVAE Alone: Achieved only 31% recall at the 1% threshold.
  • CVAE + Erosion: Recall increased to 56%.
  • Full Hybrid Pipeline (CVAE + Erosion + Pectoral Analysis):
    • At the 1% threshold: 61% recall.
  • At the 5% threshold: 83% recall.

While the system was nearly perfect at identifying "incorrect exposure" (AUPRC of 1.0), it found "improper radiography" (AUPRC of 0.066) much more challenging due to the subtle nature of positioning errors.

Discussion and Insights

The study highlights several critical lessons for medical data curation. First, the "normal" class in mammography is inherently heterogeneous, containing both healthy tissue and various pathologies. This diversity makes it difficult for a VAE to establish a rigid boundary for what constitutes a standard image.

Second, the research demonstrates that deep learning is not a universal solution. While the CVAE was adept at finding large anomalies like breast implants, it was surprisingly insensitive to smaller artifacts. The significant jump in recall provided by simple morphological erosion suggests that hybrid models—combining the representational power of neural networks with the precision of hand-crafted rules—are currently the most robust approach for clinical data cleaning.

Finally, the study found that analyzing the latent space (via OCSVM) often yielded better results than simple reconstruction loss. This suggests that the way an outlier is positioned within the model's internal understanding of the data is more telling than just the "blurriness" of its reconstruction.

Conclusion

This hybrid pipeline provides a scalable solution for cleaning massive datasets like ADMANI. By automating the detection of technical outliers, researchers can significantly reduce the risk of model confounding. Future work will explore the use of Vector Quantised-VAE (VQ-VAE) and "likelihood regret" scores to further refine the sensitivity of the deep learning components, eventually moving toward a fully supervised model as more labeled data becomes available.

Summary updated

Abstract

The ADMANI datasets (annotated digital mammograms and associated non-image datasets) from the Transforming Breast Cancer Screening with AI programme (BRAIx) run by BreastScreen Victoria in Australia are multi-centre, large scale, clinically curated, real-world databases. The datasets are expected to aid in the development of clinically relevant Artificial Intelligence (AI) algorithms for breast cancer detection, early diagnosis, and other applications. To ensure high data quality, technical outliers must be removed before any downstream algorithm development. As a first step, we randomly select 30,000 individual mammograms and use Convolutional Variational Autoencoder (CVAE), a deep generative neural network, to detect outliers. CVAE is expected to detect all sorts of outliers, although its detection performance differs among different types of outliers. Traditional image processing techniques such as erosion and pectoral muscle analysis can compensate for the poor performance of CVAE in certain outlier types. We identify seven types of technical outliers: implant, pacemaker, cardiac loop recorder, improper radiography, atypical lesion/calcification, incorrect exposure parameter and improper placement. The outlier recall rate for the test set is 61% if CVAE, erosion and pectoral muscle analysis each select the top 1% images ranked in ascending or descending order according to image outlier score under each detection method, and 83% if each selects the top 5% images. This study offers an overview of technical outliers in the ADMANI dataset and suggests future directions to improve outlier detection effectiveness.
BibTeX
@misc{li2023technical,
  author = {Hui Li and Carlos A. Peña-Solórzano and Susan Wei and Davis J. McCarthy},
  title = {{Technical outlier detection via convolutional variational autoencoder for the ADMANI breast mammogram dataset}},
  year = {2023},
  eprint = {2305.12068},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.12068},
}
Figure