Deep Learning using Robust Interdependent Codes

Hugo Larochelle, Dumitru Erhan, Pascal VincentAISTATS 2009 · OpenReview

Abstract

We investigate a simple yet effective method to introduce inhibitory and excitatory interactions between units in the layers of a deep neural network classifier. The method is based on the greedy layer-wise procedure of deep learning algorithms and extends the denoising autoencoder by adding asymmetric lateral connections between its hidden coding units. This approach is computationally more efficient than previously proposed recursive methods. Experiments on character recognition problems show that lateral connections significantly improve classification performance in deep networks.

Introduction

In the early stages of deep learning research, the standard approach for building deep architectures involved greedy layer-wise pre-training using modules like Restricted Boltzmann Machines (RBMs) or basic Autoencoders. A primary limitation of these modules is the independence of hidden units: when a layer processes an input, every unit calculates its activation without considering the state of its neighbors.

In biological vision, neurons utilize "lateral inhibition" to compete, sharpening signals and resolving ambiguity between overlapping features. While previous machine learning models attempted to mimic this through recursive equations, they were often computationally expensive and difficult to optimize. This paper introduces Denoising Autoencoders with Interdependent Codes (DA-IC), a method that achieves these interactions in a single feed-forward pass.

The Interdependent Code (IC) Layer

The core innovation of this work is the addition of lateral inhibitory and excitatory interactions between hidden units during the encoding process. The DA-IC model follows a specific sequence:

  1. Pre-encoding: The input $x$ is transformed into a "pre-encoding" using standard weights $W$.
  2. Lateral Interaction: These pre-encodings are passed through a square matrix $V$ representing lateral connections. This allows units to excite or inhibit one another.
  3. One-Pass Computation: Unlike recursive models that require multiple iterations to "settle," DA-IC computes these interactions in a single step, making it as fast as a standard neural network layer.
The architecture of a Denoising Autoencoder with Interdependent Codes (DA-IC), showing the lateral connection matrix V.

By sharing weights between the encoder and decoder (using transposed weights), the model is forced to use these lateral connections to "explain away" redundancies, leading to more efficient and robust representations.

Experimental Results

The model was evaluated on two challenging character recognition benchmarks:

  • MNIST-rot: A variation of the MNIST digit dataset with random rotations. SDAIC-3 (a 3-layer stack) achieved an error rate of 8.07%, significantly outperforming standard Denoising Autoencoders (9.53%) and Deep Belief Networks (9.01%).
  • OCR-letters: A dataset of binary English character images. SDAIC-3 consistently outperformed baselines across cross-validation folds, achieving an average error of 9.56%.

Qualitative Analysis of Lateral Connections

Visualizing the learned weights provides insight into how the lateral matrix $V$ functions. The network implements a form of "explaining away": if two neurons detect similar features, the stronger one can inhibit the weaker one to prevent redundant representation.

Visualization of learned filters and the neurons positively/negatively connected by the lateral matrix V.

As shown in the visualizations, "positively connected" neurons typically have overlapping filters (detecting similar features from slightly different perspectives), while "negatively connected" neurons detect mutually exclusive features, such as a specific pen stroke versus an empty background. This results in hidden units that are significantly less correlated than those in standard autoencoders, suggesting a richer feature set.

Conclusion

DA-IC demonstrates that lateral interactions can be integrated into deep networks without the computational burden of recursive inference. By allowing neurons to communicate within a layer, the network learns to resolve visual overlaps more effectively, leading to state-of-the-art performance on complex character recognition tasks. Future work may explore the benefits of such interdependent codes in generative tasks or larger-scale natural image datasets.

Summary updated

Abstract

We investigate a simple yet effective method to introduce inhibitory and excitatory interactions between units in the layers of a deep neural network classifier. The method is based on the greedy layer-wise procedure of deep learning algorithms and extends the denoising autoencoder of (Vincent et al., 2008) by adding asymmetric lateral connections between its hidden coding units, in a manner that is much simpler and computationally more efficient than previously proposed approaches. We present experiments on two character recognition problems which show for the first time that lateral connections can significantly improve the classification performance of deep networks.
BibTeX
@inproceedings{larochelle2009deep,
  author = {Hugo Larochelle and Dumitru Erhan and Pascal Vincent},
  title = {{Deep Learning using Robust Interdependent Codes}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2009},
  url = {https://openreview.net/forum?id=bxY419SgAF},
}
Figure