An Information Maximization Approach to Overcomplete and Recurrent Representations

Oren Shriki, Haim Sompolinsky, Daniel D. LeeNeurIPS 2000 · OpenReview

Introduction

This paper generalizes the "Infomax" principle—a cornerstone of Independent Component Analysis (ICA)—to neural networks that are both overcomplete (having more outputs than inputs) and recurrent. By maximizing the entropy of the output units, the authors derive simple, biologically plausible learning rules that allow networks to discover complex features and implement automatic gain control through feedback.

The Challenge of Complex Architectures

In unsupervised learning, we often want a network to "understand" its input by transforming it into a useful internal representation. Traditional ICA usually assumes a "square" architecture: if you have 10 input sensors, you have 10 output features.

However, both biological brains and modern machine learning often benefit from overcomplete representations, where the number of features exceeds the number of inputs. This allows for more robust and sparse coding. Furthermore, real neural circuits are rarely just feedforward; they are packed with recurrent (feedback) connections.

The challenge addressed here is how to mathematically derive learning rules for these complex architectures using information theory. Specifically, how do we maximize the "mutual information" between the world (input) and the network's internal state (output) when the mapping is non-linear, overcomplete, and recursive?

Network Architecture and Methods

The authors propose a network model where $N$ inputs drive $M$ outputs ($M \ge N$). The outputs are determined by a non-linear "squashing" function (like tanh) that takes both the feedforward input (weighted by matrix $W$) and recurrent feedback from other outputs (weighted by matrix $K$).

Network Architecture

To derive the learning rules, the authors employ several key methodological steps:

  • The Zero-Noise Limit: By assuming the network is deterministic (no noise), the mutual information between input and output becomes equivalent to the entropy of the output distribution.
  • Manifold Mapping: Because the output space is larger than the input space, the outputs actually live on a lower-dimensional "manifold" embedded in a high-dimensional space. The authors use the Jacobian (the "susceptibility matrix") to measure how volume elements in the input space are stretched or compressed when mapped to this output manifold.
  • Gradient Descent and Natural Gradient: They derive specific update rules for both the feedforward weights ($W$) and recurrent weights ($K$) by calculating the gradient of the output entropy. Applying a "natural gradient" trick makes the learning rules more efficient and demonstrates that their math perfectly reduces to the classic Bell-Sejnowski ICA algorithm when the network is square and non-recurrent.

Experimental Validation: The 2D Hexagonal Distribution

To illustrate the concepts, the paper focuses on a synthetic 2D hexagonal distribution. This two-dimensional input ($N=2$) is generated from a linear combination of three vectors separated by 120 degrees, creating a hexagonal cloud of data points containing higher-order correlations that standard linear methods (like PCA) would miss.

Using the derived rules, a network with 3 output units successfully converged to the optimal orientations within about 20 iterations.

Learned Filters on Hexagonal Data

As shown above, the network learned three filters that aligned with the symmetries of the input. While the relative orientations (angles) between filters remained consistent (e.g., 120 degrees apart), the network could settle into different "local minima" depending on initial conditions, reflecting the inherent symmetry of the problem.

Key Insights

  • Recurrence as Gain Control: Perhaps the most striking insight is that optimizing recurrent connections ($K$) performs automatic gain control. If the input signals are weak, the recurrent weights become negative to amplify the signal. If inputs are strong, the weights become positive to suppress the signal and prevent the output units from saturating.
  • Overcompleteness and Feature Selection: In the overcomplete case, the network doesn't just find "directions" in the data; it finds a set of filters that best span the input distribution.
  • Unified Framework: The paper provides a single mathematical framework that handles square, overcomplete, and even undercomplete representations just by changing the order of matrix products in the cost function.

Limitations and Future Work

While the rules work beautifully for simple 2D examples, the authors acknowledge that more research is needed to understand how they scale to high-dimensional, real-world data where speed of convergence and local minima might become significant hurdles. Additionally, the study used a fixed tanh nonlinearity; future work could involve "learning" the optimal nonlinearity to better match specific input statistics, or adaptively adding/removing output units to find the perfect level of overcompleteness.

Summary updated

Abstract

The principle of maximizing mutual information is applied to learning overcomplete and recurrent representations. The underlying model con(cid:173) sists of a network of input units driving a larger number of output units with recurrent interactions. In the limit of zero noise, the network is de(cid:173) terministic and the mutual information can be related to the entropy of the output units. Maximizing this entropy with respect to both the feed(cid:173) forward connections as well as the recurrent interactions results in simple learning rules for both sets of parameters. The conventional independent components (ICA) learning algorithm can be recovered as a special case where there is an equal number of output units and no recurrent con(cid:173) nections. The application of these new learning rules is illustrated on a simple two-dimensional input example.
BibTeX
@inproceedings{shriki2000information,
  author = {Oren Shriki and Haim Sompolinsky and Daniel D. Lee},
  title = {{An Information Maximization Approach to Overcomplete and Recurrent Representations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2000},
  url = {https://openreview.net/forum?id=H1bNXDW_br},
}
Figure