Abstract
We investigate a simple yet effective method to introduce inhibitory and excitatory interactions between units in the layers of a deep neural network classifier. The method is based on the greedy layer-wise procedure of deep learning algorithms and extends the denoising autoencoder by adding asymmetric lateral connections between its hidden coding units. This approach is computationally more efficient than previously proposed recursive methods. Experiments on character recognition problems show that lateral connections significantly improve classification performance in deep networks.
Introduction
In the early stages of deep learning research, the standard approach for building deep architectures involved greedy layer-wise pre-training using modules like Restricted Boltzmann Machines (RBMs) or basic Autoencoders. A primary limitation of these modules is the independence of hidden units: when a layer processes an input, every unit calculates its activation without considering the state of its neighbors.
In biological vision, neurons utilize "lateral inhibition" to compete, sharpening signals and resolving ambiguity between overlapping features. While previous machine learning models attempted to mimic this through recursive equations, they were often computationally expensive and difficult to optimize. This paper introduces Denoising Autoencoders with Interdependent Codes (DA-IC), a method that achieves these interactions in a single feed-forward pass.
The Interdependent Code (IC) Layer
The core innovation of this work is the addition of lateral inhibitory and excitatory interactions between hidden units during the encoding process. The DA-IC model follows a specific sequence:
- Pre-encoding: The input $x$ is transformed into a "pre-encoding" using standard weights $W$.
- Lateral Interaction: These pre-encodings are passed through a square matrix $V$ representing lateral connections. This allows units to excite or inhibit one another.
- One-Pass Computation: Unlike recursive models that require multiple iterations to "settle," DA-IC computes these interactions in a single step, making it as fast as a standard neural network layer.
By sharing weights between the encoder and decoder (using transposed weights), the model is forced to use these lateral connections to "explain away" redundancies, leading to more efficient and robust representations.
Experimental Results
The model was evaluated on two challenging character recognition benchmarks:
- MNIST-rot: A variation of the MNIST digit dataset with random rotations. SDAIC-3 (a 3-layer stack) achieved an error rate of 8.07%, significantly outperforming standard Denoising Autoencoders (9.53%) and Deep Belief Networks (9.01%).
- OCR-letters: A dataset of binary English character images. SDAIC-3 consistently outperformed baselines across cross-validation folds, achieving an average error of 9.56%.
Qualitative Analysis of Lateral Connections
Visualizing the learned weights provides insight into how the lateral matrix $V$ functions. The network implements a form of "explaining away": if two neurons detect similar features, the stronger one can inhibit the weaker one to prevent redundant representation.
As shown in the visualizations, "positively connected" neurons typically have overlapping filters (detecting similar features from slightly different perspectives), while "negatively connected" neurons detect mutually exclusive features, such as a specific pen stroke versus an empty background. This results in hidden units that are significantly less correlated than those in standard autoencoders, suggesting a richer feature set.
Conclusion
DA-IC demonstrates that lateral interactions can be integrated into deep networks without the computational burden of recursive inference. By allowing neurons to communicate within a layer, the network learns to resolve visual overlaps more effectively, leading to state-of-the-art performance on complex character recognition tasks. Future work may explore the benefits of such interdependent codes in generative tasks or larger-scale natural image datasets.
Summary updated