Developing Population Codes by Minimizing Description Length

Richard S. Zemel, Geoffrey E. HintonNeurIPS 1993 · OpenReview

Introduction

In unsupervised learning, many algorithms aim to produce "factorial codes," where each hidden unit represents an independent feature of the data. While efficient, these codes are fragile—if one unit fails, information is lost entirely. Furthermore, they do not resemble the highly redundant "population codes" found in biological brains.

Conversely, existing topographic algorithms, like Kohonen Self-Organizing Maps (SOMs), create representations where nearby units respond to similar inputs. However, these maps usually rely on a fixed topology defined by the researcher in advance. This paper introduces a method for training neural networks to develop robust, biologically inspired population codes where the network learns its own representation space. By applying the Minimum Description Length (MDL) principle, the authors demonstrate that a network can organize its hidden units into an "implicit space" that efficiently encodes complex, non-linear input data.

The Minimum Description Length (MDL) Approach

The authors utilize an autoencoder framework governed by the MDL principle. The core idea of MDL is that the optimal representation is the one that minimizes the total "cost" (measured in bits) of communicating the input to a receiver.

This total cost comprises three main components:

  1. Code-cost: The cost of describing the hidden unit activities.
  2. Model-cost: The cost of describing the network weights (which is ignored in this specific study).
  3. Reconstruction-error: The cost of correcting any mistakes made when recreating the input from the code.

The "Implicit Space" Innovation

Instead of communicating every hidden unit's activity separately, the authors propose a more economical strategy. Each hidden unit is assigned a coordinate in a low-dimensional implicit space. The network is then heavily encouraged to make the hidden unit activities form a Gaussian "bump" within this space.

Network architecture and the mapping of hidden unit activities to a best-fit Gaussian bump in implicit space.

If the activities form a perfect Gaussian bump, the system only needs to communicate the center of that bump to describe the behavior of the entire population. The network simultaneously learns both the weights (to generate the activities) and the implicit coordinates of the units (to define the map) via backpropagation.

Experimental Setup

The model was tested on several synthetic, real-valued image datasets to evaluate its capacity to learn underlying constraint spaces:

  • 8x8 "Retina" Images: Simple shapes placed at random $(x, y)$ positions.
  • Oriented Shapes: Images where a shape varies in both position and orientation.
  • Horizontal/Vertical Bars: A dataset containing either a horizontal or vertical bar in random positions, designed to test if the network can handle "discontinuous" data (two distinct classes of shapes).

Key Insights and Results

Learning the Map and Discontinuous Topography

Unlike SOMs, where the relationship between units is a fixed grid, this model allows units to dynamically move their coordinates in the implicit space. This flexibility allows the network to discover the true dimensionality and structure of the data.

Because the coordinates are learned, the network can even "shatter" the map to handle discontinuous data. In the horizontal/vertical bar task, units representing horizontal bars clustered in one area of the implicit space, while units for vertical bars clustered in another. This effectively squeezed a categorical third dimension (shape type) into a 2D map.

Emergent Receptive Fields

The "bump-error"—the misfit between actual activity and a perfect Gaussian—acts as a powerful regularizer. It forces the network to find a representation that is "cheap" to describe, naturally leading to smooth, localized receptive fields.

Evolution of a hidden unit's receptive field from random noise at Epoch 0 to a localized response at Epoch 23.

In the $(x, y)$ position task, hidden units that started with random coordinates and weights successfully organized themselves. As training progressed, each unit developed a localized receptive field, responding only to shapes in a specific part of the image corresponding to its learned coordinate.

Hyperacuity and Dimensionality Extraction

Population codes allow the network to represent positions in the implicit space with much higher precision than the raw number of units would suggest, a property known as hyperacuity. Furthermore, the network successfully utilized a 3D implicit space to encode both position and orientation simultaneously, allowing for highly accurate reconstructions of the input images.

Limitations and Future Directions

The primary limitation of this approach is that the user must still predefine the dimensionality of the implicit space (e.g., choosing a 2D vs. 3D space) before training begins.

To address this, the authors suggest a future extension where the cost of specifying the coordinates themselves is included in the description length. By penalizing the variance of unneeded coordinates, the network could potentially "shut down" extra dimensions, automatically learning the correct intrinsic dimensionality of the data. Additionally, this MDL-based population coding approach could be extended to hierarchical, multi-layer networks to extract increasingly higher-order properties of complex inputs.

Summary updated

Abstract

The Minimum Description Length principle (MDL) can be used to train the hidden units of a neural network to extract a representa(cid:173) tion that is cheap to describe but nonetheless allows the input to be reconstructed accurately. We show how MDL can be used to develop highly redundant population codes. Each hidden unit has a location in a low-dimensional implicit space. If the hidden unit activities form a bump of a standard shape in this space, they can be cheaply encoded by the center ofthis bump. So the weights from the input units to the hidden units in an autoencoder are trained to make the activities form a standard bump. The coordinates of the hidden units in the implicit space are also learned, thus allow(cid:173) ing flexibility, as the network develops a discontinuous topography when presented with different input classes. Population-coding in a space other than the input enables a network to extract nonlinear higher-order properties of the inputs. Most existing unsupervised learning algorithms can be understood using the Min(cid:173) imum Description Length (MDL) principle (Rissanen, 1989). Given an ensemble of input vectors, the aim of the learning algorithm is to find a method of coding each input vector that minimizes the total cost, in bits, of communicating the input vectors to a receiver. There are three terms in the total description length: • The code-cost is the number of bits required to communicate the code that the algorithm assigns to each input vector.
BibTeX
@inproceedings{zemel1993developing,
  author = {Richard S. Zemel and Geoffrey E. Hinton},
  title = {{Developing Population Codes by Minimizing Description Length}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1993},
  url = {https://openreview.net/forum?id=r1WMlY-u-H},
}
Figure