Abstract
This paper addresses the "scaling problem" in early connectionist models, where the convergence time of Backpropagation (BP) increases exponentially with the number of hidden layers. Ballard proposes a modular, hierarchical architecture utilizing autoassociative modules and a "Winner Gets More" (WGM) heuristic. The results demonstrate that deep hierarchical systems can achieve convergence rates comparable to shallow three-layer networks, effectively bypassing the credit assignment delays typical of standard feedforward architectures.
Introduction: The Scaling Problem
In the late 1980s, the rediscovery of Backpropagation sparked significant interest in neural networks. However, researchers quickly encountered a fundamental limitation: the scaling problem. While standard feedforward networks could learn complex mappings, adding internal (hidden) layers to capture higher-order regularities caused convergence times to explode.
Ballard (1987) quantified this bottleneck, noting that a simple three-layer network might converge in 256 iterations, whereas a five-layer network could require over 4,000 iterations—a 20-fold increase in training time for a marginal increase in depth. This "credit assignment" problem made it computationally prohibitive to build the massively parallel, multi-level hierarchies observed in biological systems. The goal of this research was to develop a method for building deep systems without incurring this massive training penalty.
Modular Hierarchical Architecture
To solve the scaling issue, Ballard introduced a Modular Hierarchical Architecture designed to keep error propagation distances short and local. The system is built upon three primary pillars:
- Autoassociation: Instead of a direct input-to-output mapping, the network is composed of modules that attempt to reconstruct their own input (Input $\rightarrow$ Internal $ ightarrow$ Output, where Output $\approx$ Input). This creates a "virtual copy" that generates local error signals, allowing each module to learn its own internal representation somewhat independently.
- Hierarchical Coupling: Modules are stacked into hierarchies. For instance, a "Sensory" module and a "Motor" module process raw data, and their respective internal representations are then fed as input to a higher-level "Abstract" module.
- Error-Modulated Feedback: A coupling factor ($\gamma$) regulates the influence of higher-level modules on lower-level ones. When a high-level module has a high error, its influence is minimized. As it learns and its error decreases, it begins to "guide" the lower modules, ensuring that early-stage noise in the abstract layer does not corrupt the learning process in peripheral modules.
The "Winner Gets More" (WGM) Heuristic
A critical innovation in this paper is the Winner Gets More (WGM) heuristic. Standard BP treats all units uniformly, but Ballard modified the weight update rule so that weight changes are scaled by the relative activation of the unit. This encourages "sparse encodings," where only a few units are active for any given pattern. By reducing interference between different learned patterns, WGM significantly accelerates the gradient descent process.
Experimental Evaluation: The Encoder Problem
The proposed architecture was evaluated using the Encoder Problem, a benchmark requiring a network to pass a 4-bit input (where exactly one bit is "on") through a bottleneck layer and reconstruct it perfectly at the output. Ballard compared three configurations:
- A standard 4-4-4 (3-layer) network.
- A standard 4-4-4-4-4 (5-layer) network.
- A modular system consisting of two 4-4-4 modules feeding into one 8-8-8 abstract module.
Results and Scaling Efficiency
The experiments confirmed the severity of the scaling wall: the 5-layer standard BP network took roughly 20 times longer to converge than the 3-layer network. In contrast, the Modular System converged in a time comparable to the simple 3-layer network, despite having twice as many connections and a more complex hierarchy.
By decoupling the modules through autoassociation and using WGM to enforce sparsity, the network effectively broke the symmetry that slows down standard BP. The error-modulated coupling ($\gamma$) proved stable, preventing the abstract layer from interfering with lower-level learning during the initial phases of training.
Conclusion and Limitations
Ballard’s work demonstrated that modularity is a viable path toward scaling deep neural networks. However, the approach introduces an "inference gap": during standard use, the abstract layer may only receive partial input (e.g., sensory data without motor data), requiring the system to rely on pattern completion. While the experiments were limited to the 4-bit encoder problem, the principles of local error signals and sparse representations remain foundational concepts in modern attempts to build efficient, deep hierarchical models.
Summary updated