Processsing Simple Geometric Attributes with Autoencoders

Alasdair Newson, Andrés Almansa, Yann Gousseau, Saïd LadjalarXiv 2019 · arXiv

Introduction

While deep learning has achieved remarkable success in generating complex data like high-fidelity faces or landscapes, the fundamental mechanisms by which neural networks process basic geometric attributes remain poorly understood. In "Processing Simple Geometric Attributes with Autoencoders," the authors take a "back-to-basics" approach, stripping away complexity to investigate how simple autoencoders (AEs) handle the building blocks of vision: size and position. By using toy datasets and a single-dimensional latent space, this research provides both theoretical and experimental insights into the internal mechanics of generative models.

The Problem of Interpretability

Deep generative models, such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), are powerful but often function as "black boxes." They are prone to issues like "mode collapse" and struggle to generalize to data slightly outside their training distribution. The authors argue that addressing these high-level challenges requires a foundational understanding of how a network maps a pixel-grid image to a single scalar attribute in latent space and back again.

Methodology

The researchers employed a standard Convolutional Autoencoder architecture featuring an extreme "bottleneck"—a latent space of exactly one dimension ($d=1$). This design forces the network to compress all information about the input image into a single number $z$.

Architecture of the convolutional autoencoder with a 1D bottleneck.

Key methodological components included:

  • Ablation Studies: Systematically removing network biases to isolate their functional role.
  • Theoretical Derivation: For the bias-free case, the authors derived a closed-form solution for the energy minimized during training, proving convergence to a specific radially symmetric function.
  • Regularization Analysis: Three types of regularization were compared to evaluate their impact on generalization (filling "holes" in missing data): standard weight decay on both encoder and decoder, locality-preservation, and asymmetric weight decay applied only to the encoder.

Datasets

To maintain mathematical clarity, the study utilized synthetic datasets:

  • Centered Disks: 3,000 grey-level images of disks with radii sampled uniformly. Disks were slightly blurred with a Gaussian filter to ensure a continuous data manifold.
  • 1D Dirac Deltas: "One-hot" vectors used to analyze how the network extracts and reconstructs spatial position.

Key Insights: Area, Bias, and Regularization

The study yielded several critical insights into the behavior of convolutional networks:

Area over Radius: The network naturally learns to represent the area of a disk rather than its radius. Because convolution is essentially an integration process, the sum of pixels in a binary disk—its area—is the most direct mapping to the latent variable $z$.

Decoded disks and the non-linear relationship between radius $r$ and latent variable $z$.

The Necessity of Bias: Without biases, the decoder can only scale a single base function multiplicatively. Biases are essential for creating the "thresholding" effect required to produce sharp, distinct edges for different disk sizes.

Asymmetric Regularization for Generalization: A significant finding was that regularizing only the encoder produced the smoothest latent space and the best generalization. This constraint forces the encoder to remain "simple," which in turn compels the decoder to learn a more robust mapping capable of interpolating across missing data ranges.

Results and Discussion

The empirical results closely aligned with the authors' theoretical predictions. Specifically, the decoder's behavior in the bias-free case was found to match a closed-form solution involving Airy functions. Furthermore, the network demonstrated an innate ability to discover optimal filters for position encoding, such as the $[1, 2, 1]$ filter combined with subsampling. Most importantly, the asymmetric regularization allowed the model to successfully reconstruct disks of sizes it had never encountered during training, effectively "filling the hole" in the data manifold.

Conclusion

This research highlights the value of studying simplified models to uncover the fundamental principles of deep learning. While the study is currently limited to 1D and 2D toy shapes, it provides a rigorous framework for understanding attribute encoding. Future work will explore how these mechanisms scale to more complex attributes like rotation and color, and whether asymmetric regularization remains effective in larger, multi-attribute generative architectures.

Summary updated

Abstract

Image synthesis is a core problem in modern deep learning, and many recent architectures such as autoencoders and Generative Adversarial networks produce spectacular results on highly complex data, such as images of faces or landscapes. While these results open up a wide range of new, advanced synthesis applications, there is also a severe lack of theoretical understanding of how these networks work. This results in a wide range of practical problems, such as difficulties in training, the tendency to sample images with little or no variability, and generalisation problems. In this paper, we propose to analyse the ability of the simplest generative network, the autoencoder, to encode and decode two simple geometric attributes : size and position. We believe that, in order to understand more complicated tasks, it is necessary to first understand how these networks process simple attributes. For the first property, we analyse the case of images of centred disks with variable radii. We explain how the autoencoder projects these images to and from a latent space of smallest possible dimension, a scalar. In particular, we describe a closed-form solution to the decoding training problem in a network without biases, and show that during training, the network indeed finds this solution. We then investigate the best regularisation approaches which yield networks that generalise well. For the second property, position, we look at the encoding and decoding of Dirac delta functions, also known as `one-hot' vectors. We describe a hand-crafted filter that achieves encoding perfectly, and show that the network naturally finds this filter during training. We also show experimentally that the decoding can be achieved if the dataset is sampled in an appropriate manner.
BibTeX
@misc{newson2019processsing,
  author = {Alasdair Newson and Andrés Almansa and Yann Gousseau and Saïd Ladjal},
  title = {{Processsing Simple Geometric Attributes with Autoencoders}},
  year = {2019},
  eprint = {1904.07099},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.07099},
}
Figure