Taking Apart Autoencoders: How do They Encode Geometric Shapes ?

Alasdair Newson, Andrés Almansa, Yann Gousseau, Saïd LadjalHAL 2018

TL;DR

This paper dissects the inner workings of autoencoders by studying their ability to encode and decode simple geometric shapes (disks). They reveal that autoencoders can learn to represent disks efficiently, but struggle with generalization when faced with data outside their training distribution. The authors also explore regularization techniques to improve this generalization capability.

What Problem Does This Paper Address?

Autoencoders are powerful tools for dimensionality reduction and feature learning, widely used in tasks like image compression and generation. However, despite their prevalence, the precise mechanisms by which they learn and represent data remain largely a "black box." This paper aims to demystify these inner workings by taking a controlled, in-depth approach to understanding how an autoencoder encodes and decodes a simple geometric shape – a disk. This controlled setting allows for a more rigorous analysis of the autoencoder's behavior, particularly its ability to generalize to unseen data, a known challenge in deep learning.

Methods

The authors employ a convolutional autoencoder architecture, specifically designed for processing centered disks. They meticulously analyze the autoencoder's behavior under various conditions, including:

  • Optimal Encoding: They investigate the theoretical optimal solution for encoding a disk and show that their autoencoder approximates it.
  • Ablation Study (No Biases): By removing biases from the decoder, they isolate the core function learned for decoding disks and derive its mathematical form.
  • Generalization Experiments: They systematically remove specific disk radii from the training data to test the autoencoder's ability to interpolate and extrapolate to unseen radii.
  • Regularization Techniques: They experiment with several regularization methods (e.g., latent space regularization, weight penalization) to improve the autoencoder's generalization performance.

Datasets

The primary dataset used is a collection of binary images of centered disks with varying radii. This dataset is artificially constructed for controlled experiments.

Key Insights and Ideas

  • Encoder as Integration: The encoder, when trained on disks, appears to learn to integrate over the disk's area, effectively encoding the disk's area as the latent representation.
  • Decoder as Scaled Function: Without biases, the decoder learns a fundamental radial profile function that is then scaled by a factor dependent on the input disk's radius. Biases are crucial for inducing a thresholding mechanism to create a sharp disk boundary.
  • Generalization Failure: A significant finding is the autoencoder's inability to generalize beyond the range of radii seen during training. Instead of extrapolating, it tends to project unseen disks onto the closest observed radii, indicating a lack of true abstraction.
  • Regularization's Impact: Certain regularization techniques, particularly an asymmetrical approach that penalizes encoder weights while leaving the decoder more freedom, show promise in improving generalization by creating a more continuous latent space.

Key Results

  • The autoencoder successfully approximates the theoretical optimal solution for encoding disks.
  • When trained without biases, the decoder learns a function that, when multiplied by a radius-dependent scalar, accurately reconstructs disks.
  • The autoencoder fails to extrapolate to disk radii not present in the training data, even when compared to more complex architectures like iGAN.
  • Asymmetrical regularization (Type 3) significantly improves the autoencoder's ability to handle "holes" in the training data, leading to better reconstruction of unseen disk radii.

Limitations and Future Work

  • Limited Scope: The study is confined to a very simple geometric shape (disks). While this allows for deep analysis, it raises questions about whether these mechanisms generalize to more complex objects.
  • Extrapolation vs. Interpolation: The paper focuses on the failure to extrapolate. Further work could explore the nuances of interpolation within observed ranges.
  • Decoder with Biases: The paper notes that the behavior of the decoder with biases included requires further study, especially as the radius increases.
  • Extending to Complex Objects: A key future goal is to extend these theoretical analyses to increasingly complex visual objects to understand if similar encoding mechanisms persist and to develop better regularization schemes for complex networks.

Summary updated

Abstract

An extended version of this preprint <hal-02271281> was published at JMIV 2019 under doi:10.1007/s10851-019-00924-w
BibTeX
@inproceedings{newson2018taking,
  author = {Alasdair Newson and Andrés Almansa and Yann Gousseau and Saïd Ladjal},
  title = {{Taking Apart Autoencoders: How do They Encode Geometric Shapes ?}},
  booktitle = {HAL},
  year = {2018},
}
Figure