Information Potential Auto-Encoders

Yan Zhang, Mete Ozay, Zhun Sun, Takayuki OkataniarXiv 2017 · OpenReview

Abstract

In this paper, we suggest a framework to make use of mutual information as a regularization criterion to train Auto-Encoders (AEs). In the proposed framework, AEs are regularized by minimization of the mutual information between input and encoding variables of AEs during the training phase. In order to estimate the entropy of the encoding variables and the mutual information, we propose a non-parametric method. We also give an information theoretic view of Variational AEs (VAEs), which suggests that VAEs can be considered as parametric methods that estimate entropy. Experimental results show that the proposed non-parametric models have more degree of freedom in terms of representation learning of features drawn from complex distributions such as Mixture of Gaussians, compared to methods which estimate entropy using parametric approaches, such as Variational AEs.

Introduction

Auto-Encoders (AEs) are fundamental tools in unsupervised representation learning, designed to compress input data into a latent space and reconstruct it. However, high-capacity AEs often suffer from the "identity mapping" problem, where the model simply copies inputs to outputs without learning meaningful underlying structures. To prevent this, regularization is required.

A mathematically principled approach to regularization involves minimizing the mutual information between the input data and the latent representation. This forces the network to discard noise and retain only the most essential features. While Variational Auto-Encoders (VAEs) address this by assuming a parametric prior (typically a standard Gaussian), this assumption can be overly restrictive. Real-world data often exhibits multi-modal distributions that a single Gaussian cannot capture, leading to the destruction of structural information like distinct clusters during the compression process.

Methodology

This paper introduces Information Potential Auto-Encoders (IPAEs), which frame AE training as a rate-distortion problem: minimizing mutual information (compression) while maintaining reconstruction error (distortion) below a specific threshold.

Non-Parametric Entropy Estimation

Unlike VAEs, which rely on the Kullback-Leibler (KL) divergence against a fixed prior, IPAEs utilize a non-parametric method to estimate the entropy of the latent space:

  1. Stochastic Encoding: The encoder generates a mean and variance, and latent vectors are sampled using the re-parameterization trick.
  2. Empirical Entropy Approximation: The entropy of the latent space is estimated by averaging the conditional distributions of a batch of samples, rather than comparing them to a pre-defined distribution.
  3. Information Potential: By applying Jensen’s inequality to the empirical distribution, the regularization term is transformed into a function of pairwise distances between samples in the latent space.

Geometric Interpretation

This approach introduces a different geometric dynamic than VAEs. While VAEs exert a "global gravitational pull" that drags all samples toward the origin (0) of the latent space, IPAEs create a localized clustering force. In this "information potential" framework, pairs of samples are pulled toward each other. This provides the latent space with more degrees of freedom, allowing it to naturally form multiple distinct clusters without being forced into a unimodal Gaussian shape.

Experimental Results

Toy Dataset: Mixture of Gaussians

The models were tested on a synthetic 2D dataset consisting of 5,000 samples from 25 distinct Gaussian distributions. IPAEs successfully preserved all 25 clusters in the latent space while filtering noise, achieving a reconstruction error of 0.00204. In contrast, VAEs either failed to regularize effectively or collapsed the clusters into a single mass at higher regularization strengths, with a best reconstruction error of 0.00728.

MNIST Classification

To evaluate the quality of learned features, a linear SVM was trained on the latent representations of an MNIST subset (digits 1, 3, and 4). IPAEs produced more linearly discriminative features, achieving a classification error of 0.70% ± 0.15%, outperforming the best VAE (0.82% ± 0.14%). Furthermore, IPAEs demonstrated superior training stability; while VAEs occasionally suffered from exploding variances at low regularization weights, IPAEs remained stable throughout training.

Discussion and Limitations

While the IPAE objective involves pairwise interactions, which could theoretically increase computational complexity, the authors found that using small sample sizes for the approximation (even $N_j = 1$) is sufficient in practice. This ensures that training remains efficient, while inference speed remains identical to standard AEs or VAEs.

Conclusion

Information Potential Auto-Encoders offer a flexible, non-parametric alternative to VAEs. By replacing global parametric priors with pairwise information potentials, IPAEs allow for richer representation learning, particularly in datasets with complex, multi-modal structures. Future work may extend this non-parametric regularization to other generative frameworks, such as Generative Adversarial Networks (GANs), to further enhance feature extraction and stability.

Summary updated

Abstract

In this paper, we suggest a framework to make use of mutual information as a regularization criterion to train Auto-Encoders (AEs). In the proposed framework, AEs are regularized by minimization of the mutual information between input and encoding variables of AEs during the training phase. In order to estimate the entropy of the encoding variables and the mutual information, we propose a non-parametric method. We also give an information theoretic view of Variational AEs (VAEs), which suggests that VAEs can be considered as parametric methods that estimate entropy. Experimental results show that the proposed non-parametric models have more degree of freedom in terms of representation learning of features drawn from complex distributions such as Mixture of Gaussians, compared to methods which estimate entropy using parametric approaches, such as Variational AEs.
BibTeX
@misc{zhang2017information,
  author = {Yan Zhang and Mete Ozay and Zhun Sun and Takayuki Okatani},
  title = {{Information Potential Auto-Encoders}},
  year = {2017},
  eprint = {1706.04635},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1706.04635},
}
Figure