Autoencoder and Partially Impossible Reconstruction Losses

Steve Dias Da Cruz, Bertram Taetz, Thomas Stifter, Didier Stricker

TL;DR

This paper introduces "Partially Impossible Reconstruction Losses" (PIRL), a novel training strategy for autoencoders that improves their ability to learn robust and generalizable features. By strategically sampling input-target pairs, PIRL encourages autoencoders to focus on invariant aspects of data, leading to better performance in tasks like image normalization, synthetic-to-real transfer, and uncertainty estimation, often outperforming existing methods.

What Problem Does This Paper Address?

Autoencoders are powerful tools for unsupervised learning, but their effectiveness often hinges on the reconstruction loss, which typically measures the pixel-wise difference between input and output. This can be a limited metric, especially when dealing with variations like illumination changes, different backgrounds, or synthetic-to-real domain shifts. The authors aim to improve autoencoder performance by developing new reconstruction loss functions that implicitly guide the model to learn more meaningful and invariant features, rather than just memorizing pixel details. They specifically address the challenge of making autoencoders robust to variations that are irrelevant to the downstream task, such as illumination changes, and improving their generalization capabilities, particularly from synthetic to real-world data.

Methods

The core contribution is the introduction of two variations of a "Partially Impossible Reconstruction Loss" (PIRL):

  • I-PIRL (Illumination PIRL): This variation uses pairs of images from the same scene but with different illumination or backgrounds as input-target pairs. The idea is that by forcing the autoencoder to reconstruct a slightly altered version of the input, it learns to ignore the variations (illumination, background) and focus on what remains constant (semantics).
  • II-PIRL (Class Instance PIRL): This more general variation uses images from the same class but different instances as input-target pairs. For example, if the input is an image of a person, the target could be another image of a person, but from a different scene or pose. This implicitly uses label information to guide the learning process, encouraging the model to learn invariances within a class.

These PIRL losses can be combined with other techniques, such as:

  • Triplet Loss: Used to structure the latent space, pushing similar class representations closer and dissimilar ones further apart.
  • Extractor Autoencoder (E-AE): Leverages pretrained convolutional neural networks (like VGG-11) as feature extractors. The autoencoder then learns on these extracted features, aiming to improve synthetic-to-real generalization.
  • Monte Carlo (MC) Dropout: Applied during inference to estimate model uncertainty and detect out-of-distribution samples.

Datasets

The paper utilizes a variety of datasets for training and evaluation:

  • Standard Datasets: MNIST, Fashion-MNIST, CIFAR10, GTSRB. These are used for initial analysis and comparison with existing methods.
  • SVIRO Dataset Family:
    • SVIRO: A synthetic dataset of vehicle interiors for occupant classification.
  • SVIRO-Illumination: Variations of SVIRO scenes under different lighting conditions.
  • SVIRO-NoCar: SVIRO scenes with backgrounds randomized instead of a vehicle.
  • SVIRO-Uncertainty: Fine-grained splits of SVIRO for uncertainty and OOD evaluation.
  • Extended Yale Face Database B: Used for illumination normalization experiments.
  • Webcam Clip Art Dataset: Used for demonstrating image normalization and generalization.
  • TICaM Dataset: A real-world dataset of vehicle interiors, used for synthetic-to-real transfer experiments.

Key Insights and Ideas

  • Implicit Labeling is Powerful: The II-PIRL loss, which uses label information implicitly by pairing images of the same class, achieves surprisingly good latent space representations, often outperforming the explicit triplet loss in terms of clustering and classification accuracy.
  • Focusing on Invariances: The core idea of PIRL is to force the model to learn what doesn't change between input and target pairs, effectively removing irrelevant variations like illumination or background. This leads to more robust feature extraction.
  • Synthetic-to-Real Transfer Benefits from PIRL: Combining PIRL with a pretrained feature extractor significantly improves the transfer of models trained on synthetic data to real-world scenarios, a critical challenge in robotics and autonomous systems.
  • Uncertainty Estimation is Enhanced: PIRL, especially when combined with MC dropout, leads to better separation between in-distribution and out-of-distribution samples, indicating improved uncertainty estimation.
  • Dataset Design Matters: The paper highlights the value of datasets with inherent variations (like SVIRO-Illumination) for training robust models, suggesting that careful dataset design can implicitly guide autoencoder learning.

Key Results

  • Classification Accuracy: II-PIRL consistently improved classification accuracy when using linear classifiers on latent space representations, outperforming vanilla autoencoders and often matching or exceeding triplet loss-based autoencoders on datasets like MNIST, Fashion-MNIST, CIFAR10, and GTSRB. For example, II-AE achieved 98.9% accuracy on GTSRB compared to TAE's 98.8% and AE's 95.7%.
  • Synthetic-to-Real Generalization: The II-E-TAE (Extractor Autoencoder with II-PIRL and Triplet Loss) significantly outperformed baseline models (scratch, pretrained, and E-TAE) on the TICaM dataset, achieving up to 81.0% accuracy with VGG-11, compared to scratch models around 58.5%.
  • Uncertainty Estimation: MC-II-AE showed better separation of in-distribution (Din) and out-of-distribution (Dout) samples, as measured by AUROC and Wasserstein distances, outperforming MC dropout and ensembles in many scenarios. For example, on GTSRB vs. CIFAR10, MC-II-AE achieved an AUROC of 92.6% compared to MC Dropout's 75.4%.
  • Image Normalization: PIRL effectively removed illumination and background variations from training data, leading to cleaner reconstructions and improved performance on downstream tasks.

Limitations and Future Work

  • Human Pose Preservation: The II-PIRL, while effective for general invariance, does not guarantee the preservation of human poses. The authors suggest future work could incorporate pose-specific constraints.
  • Dataset Size for Uncertainty: Performance on the SVIRO-Uncertainty dataset was good but not as strong as on MNIST, potentially due to the smaller dataset size. Adapting PIRL to larger datasets could further improve performance.
  • Synthetic Dataset Versatility: For synthetic-to-real transfer, the authors note that the reconstruction is not always perfect, and a more versatile synthetic dataset could improve performance on challenging real-world images.
  • Future Research Directions:
    • Extending PIRL with explicit constraints for pose and landmark preservation.
  • Investigating PIRL with other sampling strategies and exploring its implicit properties on latent space manifolds (e.g., disentanglement).
  • Facilitating data acquisition for scenarios requiring identical sceneries under varying conditions.
  • Developing more versatile synthetic datasets for improved Sim2Real transfer.

Summary updated

Abstract

The generally unsupervised nature of autoencoder models implies that the main training metric is formulated as the error between input images and their corresponding reconstructions. Different reconstruction loss variations and latent space regularizations have been shown to improve model performances depending on the tasks to solve and to induce new desirable properties such as disentanglement. Nevertheless, measuring the success in, or enforcing properties by, the input pixel space is a challenging endeavour. In this work, we want to make use of the available data more efficiently and provide design choices to be considered in the recording or generation of future datasets to implicitly induce desirable properties during training. To this end, we propose a new sampling technique which matches semantically important parts of the image while randomizing the other parts, leading to salient feature extraction and a neglection of unimportant details. The proposed method can be combined with any existing reconstruction loss and the performance gain is superior to the triplet loss. We analyse the resulting properties on various datasets and show improvements on several computer vision tasks: illumination and unwanted features can be normalized or smoothed out and shadows are removed such that classification or other tasks work more reliably; a better invariances with respect to unwanted features is induced; the generalization capacities from synthetic to real images is improved, such that more of the semantics are preserved; uncertainty estimation is superior to Monte Carlo Dropout and an ensemble of models, particularly for datasets of higher visual complexity. Finally, classification accuracy by means of simple linear classifiers in the latent space is improved compared to the triplet loss. For each task, the improvements are highlighted on several datasets commonly used by the research community, as well as in automotive applications.
BibTeX
@inproceedings{cruz2022autoencoder,
  author = {Steve Dias Da Cruz and Bertram Taetz and Thomas Stifter and Didier Stricker},
  title = {{Autoencoder and Partially Impossible Reconstruction Losses}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=-kPdtv78m8},
}

Loading related content...

Figure