An Information-Theoretic Framework for Fast and Robust Unsupervised Learning via Neural Population Infomax

Wentao Huang, Kechen ZhangICLR 2017 · OpenReview

Overview

Researchers have developed a new unsupervised learning framework that uses an asymptotic approximation of Shannon’s mutual information to train large-scale neural populations. By breaking the optimization into a hierarchical process, the algorithm achieves significantly faster training speeds and greater robustness than traditional methods like Independent Component Analysis (ICA) or Sparse Restricted Boltzmann Machines (SRBMs), while naturally handling complete, overcomplete, and undercomplete representations.

The Challenge of Representation Learning

In machine learning, representation learning seeks to discover the underlying structure of raw data without explicit labels. One of the most mathematically elegant approaches is the Infomax principle, which maximizes the mutual information (MI) between a system's input and its output.

However, Shannon’s MI is notoriously difficult to calculate and optimize for high-dimensional data and large neural networks. Traditional approaches like ICA often impose frustrating constraints, such as requiring the number of output neurons to exactly match the number of input dimensions. When scaling these to "overcomplete" representations—where there are more neurons than inputs, akin to the human visual cortex—the mathematics often become computationally intractable or prone to getting trapped in poor local optima.

Methods: Neural Population Infomax

To overcome these limitations, the authors introduce the Neural Population Infomax framework. The core innovation relies on a two-pronged strategy:

  1. Asymptotic Approximation: Instead of computing the exact MI formula, the framework uses an asymptotic approximation valid for large neural populations. This transforms a difficult integral into a manageable objective function involving the Fisher Information Matrix.
  2. Hierarchical Optimization: To avoid local minima, the learning process is broken into two stages:
  3. Stage 1: A linear transformation that maximizes information while filtering out noise (essentially a PCA/ZCA-like whitening step combined with a low-rank weight matrix).
  4. Stage 2: Optimizing the nonlinear "tuning functions" of the neurons.
Neural Population Infomax Architecture

By finding an analytical solution for these stages first, the method creates a "warm start" for a final gradient descent pass. This hierarchical approach ensures the model starts very close to the global optimum. Furthermore, the model incorporates biological realism, such as Poisson spiking noise and membrane noise, which acts as a natural regularizer to resist overfitting.

Key Insights

  • The "Large Population" Advantage: Typically, more neurons increase computational complexity. Here, as the population size grows, the mathematics simplify into a predictable asymptotic form, making large-scale simulation an asset.
  • Flexibility in Architecture: Unlike standard ICA, this method seamlessly handles overcomplete (e.g., 100 inputs to 1,000 outputs) and undercomplete architectures.
  • Pre-whitening is Natural: The hierarchical infomax mathematics naturally give rise to a pre-whitening step, providing theoretical justification for this common deep learning preprocessing trick.

Performance and Results

The framework was evaluated on classic benchmarks, including Olshausen’s Natural Image Dataset and MNIST, yielding impressive results:

  • Speed: The proposed algorithm converged significantly faster than existing methods, often achieving an order of magnitude (10x) speedup compared to Infomax ICA (IICA).
  • Feature Quality: The model successfully learned sharp, localized Gabor-like filters (edge detectors) from natural images. In contrast, SRBMs struggled to produce these filters as clearly under similar conditions.
  • Denoising Prowess: In image denoising tasks, the framework outperformed standard dictionary learning, achieving better reconstruction using only 14 bases than competing methods did with 100.
  • Robustness: Convergence was largely insensitive to the choice of learning rate parameters, making the model easier to deploy out-of-the-box than sensitive probabilistic alternatives.
Convergence Speed and Entropy Metrics

Limitations and Future Directions

While the paper demonstrates excellent results for single-layer feature extraction, its application to deep architectures remains primarily theoretical. The authors suggest the method can be extended to multilayer networks by stacking these infomax layers, but extensive benchmarks for very deep architectures are not provided.

Additionally, the framework relies on the assumption of a "large population" for the asymptotic mathematics to hold. While this aligns with biological brain models, its efficacy in "slim" artificial networks with few hidden units requires further investigation. Future work may explore applying this to supervised tasks to determine if Infomax pre-training provides a superior starting point for classification.

Summary updated

Abstract

A framework is presented for unsupervised learning of representations based on infomax principle for large-scale neural populations. We use an asymptotic approximation to the Shannon's mutual information for a large neural population to demonstrate that a good initial approximation to the global information-theoretic optimum can be obtained by a hierarchical infomax method. Starting from the initial solution, an efficient algorithm based on gradient descent of the final objective function is proposed to learn representations from the input datasets, and the method works for complete, overcomplete, and undercomplete bases. As confirmed by numerical experiments, our method is robust and highly efficient for extracting salient features from input datasets. Compared with the main existing methods, our algorithm has a distinct advantage in both the training speed and the robustness of unsupervised representation learning. Furthermore, the proposed method is easily extended to the supervised or unsupervised model for training deep structure networks.
BibTeX
@inproceedings{huang2017information,
  author = {Wentao Huang and Kechen Zhang},
  title = {{An Information-Theoretic Framework for Fast and Robust Unsupervised Learning via Neural Population Infomax}},
  booktitle = {International Conference on Learning Representations},
  year = {2017},
  url = {https://openreview.net/forum?id=SkYbF1slg},
}
Figure