Introduction
This paper generalizes the "Infomax" principle—a cornerstone of Independent Component Analysis (ICA)—to neural networks that are both overcomplete (having more outputs than inputs) and recurrent. By maximizing the entropy of the output units, the authors derive simple, biologically plausible learning rules that allow networks to discover complex features and implement automatic gain control through feedback.
The Challenge of Complex Architectures
In unsupervised learning, we often want a network to "understand" its input by transforming it into a useful internal representation. Traditional ICA usually assumes a "square" architecture: if you have 10 input sensors, you have 10 output features.
However, both biological brains and modern machine learning often benefit from overcomplete representations, where the number of features exceeds the number of inputs. This allows for more robust and sparse coding. Furthermore, real neural circuits are rarely just feedforward; they are packed with recurrent (feedback) connections.
The challenge addressed here is how to mathematically derive learning rules for these complex architectures using information theory. Specifically, how do we maximize the "mutual information" between the world (input) and the network's internal state (output) when the mapping is non-linear, overcomplete, and recursive?
Network Architecture and Methods
The authors propose a network model where $N$ inputs drive $M$ outputs ($M \ge N$). The outputs are determined by a non-linear "squashing" function (like tanh) that takes both the feedforward input (weighted by matrix $W$) and recurrent feedback from other outputs (weighted by matrix $K$).
To derive the learning rules, the authors employ several key methodological steps:
- The Zero-Noise Limit: By assuming the network is deterministic (no noise), the mutual information between input and output becomes equivalent to the entropy of the output distribution.
- Manifold Mapping: Because the output space is larger than the input space, the outputs actually live on a lower-dimensional "manifold" embedded in a high-dimensional space. The authors use the Jacobian (the "susceptibility matrix") to measure how volume elements in the input space are stretched or compressed when mapped to this output manifold.
- Gradient Descent and Natural Gradient: They derive specific update rules for both the feedforward weights ($W$) and recurrent weights ($K$) by calculating the gradient of the output entropy. Applying a "natural gradient" trick makes the learning rules more efficient and demonstrates that their math perfectly reduces to the classic Bell-Sejnowski ICA algorithm when the network is square and non-recurrent.
Experimental Validation: The 2D Hexagonal Distribution
To illustrate the concepts, the paper focuses on a synthetic 2D hexagonal distribution. This two-dimensional input ($N=2$) is generated from a linear combination of three vectors separated by 120 degrees, creating a hexagonal cloud of data points containing higher-order correlations that standard linear methods (like PCA) would miss.
Using the derived rules, a network with 3 output units successfully converged to the optimal orientations within about 20 iterations.
As shown above, the network learned three filters that aligned with the symmetries of the input. While the relative orientations (angles) between filters remained consistent (e.g., 120 degrees apart), the network could settle into different "local minima" depending on initial conditions, reflecting the inherent symmetry of the problem.
Key Insights
- Recurrence as Gain Control: Perhaps the most striking insight is that optimizing recurrent connections ($K$) performs automatic gain control. If the input signals are weak, the recurrent weights become negative to amplify the signal. If inputs are strong, the weights become positive to suppress the signal and prevent the output units from saturating.
- Overcompleteness and Feature Selection: In the overcomplete case, the network doesn't just find "directions" in the data; it finds a set of filters that best span the input distribution.
- Unified Framework: The paper provides a single mathematical framework that handles square, overcomplete, and even undercomplete representations just by changing the order of matrix products in the cost function.
Limitations and Future Work
While the rules work beautifully for simple 2D examples, the authors acknowledge that more research is needed to understand how they scale to high-dimensional, real-world data where speed of convergence and local minima might become significant hurdles. Additionally, the study used a fixed tanh nonlinearity; future work could involve "learning" the optimal nonlinearity to better match specific input statistics, or adaptively adding/removing output units to find the perfect level of overcompleteness.
Summary updated