Improving Image Autoencoder Embeddings with Perceptual Loss

Gustav Grund Pihlgren, Fredrik Sandin, Marcus Liwicki

TL;DR

This paper investigates how using "perceptual loss" during autoencoder training, instead of traditional pixel-wise loss, leads to better image embeddings for downstream tasks like object positioning and classification. The findings suggest that perceptual loss captures more relevant image features, making the learned representations more useful for practical applications, even if pixel-wise reconstruction quality appears worse.

What Problem Does This Paper Address?

Autoencoders are powerful tools for learning compressed representations (embeddings) of data, useful for tasks like dimensionality reduction and feature learning. Traditionally, autoencoders are trained using "element-wise loss," most commonly Mean Squared Error (MSE) applied directly to the pixel differences between the original image and its reconstruction. However, this approach has a significant drawback: it treats every pixel equally and doesn't account for the structural or semantic importance of different image regions. This can lead to embeddings that, while visually similar in terms of pixel values, fail to capture crucial high-level features. For instance, a small but important object might be poorly represented, or a shifted image might be considered "worse" by humans than a slightly blurred but structurally intact one, despite having a higher pixel-wise MSE. This paper aims to address this by exploring how "perceptual loss," which leverages a pre-trained neural network to assess image similarity in a more human-like way, can improve the quality of autoencoder embeddings for downstream tasks.

Methods

The core of this work is comparing autoencoders trained with two types of loss functions:

  • Pixel-wise Loss (MSE): The standard approach where the loss is calculated directly on the pixel differences between the input and reconstructed images.
  • Perceptual Loss: This method uses a pre-trained neural network (specifically, AlexNet in this study) as a "perceptual loss network." The loss is calculated by comparing the feature activations of this network when presented with the original image versus its reconstruction. This allows the loss to penalize differences in higher-level feature representations rather than just raw pixel values.

The authors trained both standard autoencoders (AE) and variational autoencoders (VAE) using both pixel-wise and perceptual loss on three different datasets. They then evaluated the quality of the learned embeddings by training various "predictors" (Multilayer Perceptrons and linear regressors) to perform specific tasks (object positioning or classification) using these embeddings as input. The performance of these predictors serves as the metric for the usefulness of the embeddings.

Datasets

The study utilized three datasets:

  • LunarLander-v2 collection: Images collected from the LunarLander-v2 environment. The task here is object positioning, specifically predicting the coordinates of the lander.
  • STL-10: A dataset containing unlabeled and labeled images of animals and vehicles. The task is image classification, with a focus on unsupervised learning on unlabeled data.
  • SVHN (Street View House Numbers): Images of house numbers. The task is digit classification.

Key Insights and Ideas

  • Perceptual Loss Captures More Meaningful Features: The central finding is that embeddings learned with perceptual loss are significantly better for downstream tasks than those learned with pixel-wise loss. This suggests that perceptual loss encourages the autoencoder to learn representations that are more semantically relevant.
  • Pixel-wise MSE is a Flawed Metric for Image Similarity: The paper provides strong evidence that low pixel-wise reconstruction error does not necessarily translate to useful embeddings for tasks that require understanding image content. Visualizations and results show that perceptually trained models can have higher pixel-wise reconstruction errors but still perform much better on classification and positioning tasks.
  • Embeddings Matter for Downstream Tasks: The study highlights that the quality of the autoencoder's learned embeddings is crucial for the performance of subsequent models that rely on these embeddings. This is particularly relevant for dimensionality reduction, where the goal is to retain essential information in a compact form.
  • Linear vs. Non-linear Feature Extraction: The performance difference between MLPs and linear regressors on embeddings from different loss functions reveals insights into how features are encoded. Perceptually trained embeddings might contain more detailed information that requires non-linear models to extract, whereas pixel-wise embeddings might be more amenable to linear interpretation for certain tasks.

Key Results

  • Improved Performance on Downstream Tasks: Across all three datasets and for both autoencoders and VAEs, perceptual loss consistently led to better performance in object positioning and classification tasks compared to pixel-wise loss.
  • Order of Magnitude Improvement for Object Positioning: On the LunarLander-v2 dataset, the perceptually trained autoencoders showed an "order of magnitude better performance" for object positioning compared to pixel-wise trained ones.
  • STL-10 and SVHN Classification Gains: For STL-10, perceptual loss improved MLP accuracy by up to ~20% (e.g., from ~42% to ~64% for P. VAE). For SVHN, perceptual loss also showed significant gains, with P. AE/VAE achieving ~84% accuracy compared to ~82% for AE/VAE.
  • Reconstruction vs. Task Performance Trade-off: While perceptual loss improved task performance, pixel-wise trained autoencoders often achieved lower reconstruction errors, especially on datasets like LunarLander-v2. This reinforces the idea that pixel-wise reconstruction is not the best proxy for embedding quality.
  • Increased Training Time: Perceptual loss added approximately 12% to the autoencoder training time, a trade-off considered acceptable given the performance gains.

Limitations and Future Work

  • Limited Datasets and Perceptual Loss Network: The authors acknowledge that their study used only three datasets and a single perceptual loss network (AlexNet). They suggest that further studies are needed to see if these results generalize to other datasets and different perceptual loss architectures.
  • Choice of Perceptual Loss Network and Feature Extraction: The paper used features from early layers of AlexNet. Future work could explore different layers or entirely different pre-trained networks for the perceptual loss.
  • Generalizability to Other Domains: The study focused on image tasks. Future research should investigate applying perceptual loss to other domains beyond images.
  • Computational Cost: While the increase in training time was moderate, the authors note that the cost can be significant if the perceptual loss network is very large compared to the autoencoder.
  • Linear Regressor Limitations: The significant performance drop when using linear regressors on perceptually trained embeddings for LunarLander-v2 suggests that the encoded information might require more complex models to be fully utilized. Future work could explore how to make this information more linearly accessible.

Summary updated

Abstract

Autoencoders are commonly trained using element-wise loss. However, element-wise loss disregards high-level structures in the image which can lead to embeddings that disregard them as well. A recent improvement to autoencoders that helps alleviate this problem is the use of perceptual loss. This work investigates perceptual loss from the perspective of encoder embeddings themselves. Autoencoders are trained to embed images from three different computer vision datasets using perceptual loss based on a pretrained model as well as pixel-wise loss. A host of different predictors are trained to perform object positioning and classification on the datasets given the embedded images as input. The two kinds of losses are evaluated by comparing how the predictors performed with embeddings from the differently trained autoencoders. The results show that, in the image domain, the embeddings generated by autoencoders trained with perceptual loss enable more accurate predictions than those trained with element-wise loss. Furthermore, the results show that, on the task of object positioning of a small-scale feature, perceptual loss can improve the results by a factor 10. The experimental setup is available online: https://github.com/guspih/Perceptual-Autoencoders
BibTeX
@inproceedings{pihlgren2020improving,
  author = {Gustav Grund Pihlgren and Fredrik Sandin and Marcus Liwicki},
  title = {{Improving Image Autoencoder Embeddings with Perceptual Loss}},
  booktitle = {IJCNN},
  year = {2020},
  doi = {10.1109/ijcnn48605.2020.9207431},
}
Figure