Introduction
In unsupervised learning, many algorithms aim to produce "factorial codes," where each hidden unit represents an independent feature of the data. While efficient, these codes are fragile—if one unit fails, information is lost entirely. Furthermore, they do not resemble the highly redundant "population codes" found in biological brains.
Conversely, existing topographic algorithms, like Kohonen Self-Organizing Maps (SOMs), create representations where nearby units respond to similar inputs. However, these maps usually rely on a fixed topology defined by the researcher in advance. This paper introduces a method for training neural networks to develop robust, biologically inspired population codes where the network learns its own representation space. By applying the Minimum Description Length (MDL) principle, the authors demonstrate that a network can organize its hidden units into an "implicit space" that efficiently encodes complex, non-linear input data.
The Minimum Description Length (MDL) Approach
The authors utilize an autoencoder framework governed by the MDL principle. The core idea of MDL is that the optimal representation is the one that minimizes the total "cost" (measured in bits) of communicating the input to a receiver.
This total cost comprises three main components:
- Code-cost: The cost of describing the hidden unit activities.
- Model-cost: The cost of describing the network weights (which is ignored in this specific study).
- Reconstruction-error: The cost of correcting any mistakes made when recreating the input from the code.
The "Implicit Space" Innovation
Instead of communicating every hidden unit's activity separately, the authors propose a more economical strategy. Each hidden unit is assigned a coordinate in a low-dimensional implicit space. The network is then heavily encouraged to make the hidden unit activities form a Gaussian "bump" within this space.
If the activities form a perfect Gaussian bump, the system only needs to communicate the center of that bump to describe the behavior of the entire population. The network simultaneously learns both the weights (to generate the activities) and the implicit coordinates of the units (to define the map) via backpropagation.
Experimental Setup
The model was tested on several synthetic, real-valued image datasets to evaluate its capacity to learn underlying constraint spaces:
- 8x8 "Retina" Images: Simple shapes placed at random $(x, y)$ positions.
- Oriented Shapes: Images where a shape varies in both position and orientation.
- Horizontal/Vertical Bars: A dataset containing either a horizontal or vertical bar in random positions, designed to test if the network can handle "discontinuous" data (two distinct classes of shapes).
Key Insights and Results
Learning the Map and Discontinuous Topography
Unlike SOMs, where the relationship between units is a fixed grid, this model allows units to dynamically move their coordinates in the implicit space. This flexibility allows the network to discover the true dimensionality and structure of the data.
Because the coordinates are learned, the network can even "shatter" the map to handle discontinuous data. In the horizontal/vertical bar task, units representing horizontal bars clustered in one area of the implicit space, while units for vertical bars clustered in another. This effectively squeezed a categorical third dimension (shape type) into a 2D map.
Emergent Receptive Fields
The "bump-error"—the misfit between actual activity and a perfect Gaussian—acts as a powerful regularizer. It forces the network to find a representation that is "cheap" to describe, naturally leading to smooth, localized receptive fields.
In the $(x, y)$ position task, hidden units that started with random coordinates and weights successfully organized themselves. As training progressed, each unit developed a localized receptive field, responding only to shapes in a specific part of the image corresponding to its learned coordinate.
Hyperacuity and Dimensionality Extraction
Population codes allow the network to represent positions in the implicit space with much higher precision than the raw number of units would suggest, a property known as hyperacuity. Furthermore, the network successfully utilized a 3D implicit space to encode both position and orientation simultaneously, allowing for highly accurate reconstructions of the input images.
Limitations and Future Directions
The primary limitation of this approach is that the user must still predefine the dimensionality of the implicit space (e.g., choosing a 2D vs. 3D space) before training begins.
To address this, the authors suggest a future extension where the cost of specifying the coordinates themselves is included in the description length. By penalizing the variance of unneeded coordinates, the network could potentially "shut down" extra dimensions, automatically learning the correct intrinsic dimensionality of the data. Additionally, this MDL-based population coding approach could be extended to hierarchical, multi-layer networks to extract increasingly higher-order properties of complex inputs.
Summary updated