Push it to the Limit: Discover Edge-Cases in Image Data with Autoencoders

Ilja Manakov, Volker TresparXiv 2019 · arXiv

Introduction

Deep learning models are highly sensitive to the quality of training data. In medical imaging, datasets are often aggregated from various clinical sources, leading to "garbage in, garbage out" scenarios where biased or poor-quality samples degrade model performance. Manually validating thousands of images is impractical, and existing outlier detection tools often require labels or prior knowledge of what constitutes an error. This paper presents a fully unsupervised pipeline to automatically discover semantic edge cases—such as noisy scans, improper cropping, or encoding errors—without any manual labeling.

Methodology: An Unsupervised Discovery Pipeline

The proposed method leverages the compression capabilities of autoencoders and the variance-separating power of Principal Component Analysis (PCA) in a three-step process.

The three-step pipeline: 1) Training a convolutional autoencoder, 2) Disentangling latent representations via PCA, and 3) Inspecting extreme values to identify semantic groups.
  1. Unsupervised Feature Extraction: A convolutional autoencoder is trained to reconstruct the input images. To minimize reconstruction error, the network must learn a compressed "bottleneck" representation that captures the most salient high-level features of the data.
  2. Disentanglement via PCA: Latent representations are often "tangled," where single dimensions encode multiple visual traits. Applying PCA to these representations identifies independent axes of variation, effectively disentangling different factors of image variance.
  3. Extreme Value Inspection: By sorting the dataset along these Principal Components (PCs), the authors observed that images at the extreme "tails" of the distribution (the highest and lowest values) often share distinct semantic characteristics, forming "semantic groups."

Experimental Results

The methodology was validated on three public medical imaging datasets: Brain MRI, Chest X-rays (CXR), and Optical Coherence Tomography (OCT).

Visualizing semantic groups across MRI, CXR, and OCT datasets. Extreme values of specific PCs reveal critical data quality issues like cropping, noise, and encoding errors.

Key Findings

  • Brain MRI: PC #11 successfully identified "cutoff" images. High values corresponded to scans cut off at the top, while low values highlighted scans cut off at the bottom.
  • Chest X-rays: PC #1 surfaced a critical encoding error. It identified a group of 270 images accidentally saved as RGB instead of grayscale. Due to scaling differences, these appeared nearly black, a subtle error that could easily break standard training pipelines.
  • OCT: PC #5 was found to correlate directly with the Signal-to-Noise Ratio (SNR). Images at one extreme were almost pure noise, allowing for automated quality control of the dataset.

Discussion and Conclusion

The primary advantage of this approach is its generic applicability; it requires no labels and no prior assumptions about the types of errors present in the data. Lower-order PCs typically capture broad structural features (e.g., orientation), while higher-order PCs reveal more subtle nuances.

While powerful, the method faces challenges in interpretability, as higher-order PCs can be difficult for humans to map to specific visual concepts. Future research may explore more advanced disentanglement techniques, such as $\beta$-VAEs or sparse regularization, to further refine the discovered semantic factors. Ultimately, this tool provides a scalable way for researchers to "know their data" before training begins.

Summary updated

Abstract

In this paper, we focus on the problem of identifying semantic factors of variation in large image datasets. By training a convolutional Autoencoder on the image data, we create encodings, which describe each datapoint at a higher level of abstraction than pixel-space. We then apply Principal Component Analysis to the encodings to disentangle the factors of variation in the data. Sorting the dataset according to the values of individual principal components, we find that samples at the high and low ends of the distribution often share specific semantic characteristics. We refer to these groups of samples as semantic groups. When applied to real-world data, this method can help discover unwanted edge-cases.
BibTeX
@misc{manakov2019push,
  author = {Ilja Manakov and Volker Tresp},
  title = {{Push it to the Limit: Discover Edge-Cases in Image Data with Autoencoders}},
  year = {2019},
  eprint = {1910.02713},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.02713},
}
Figure