Introduction
In the early 1990s, function interpolation in neural networks generally fell into two opposing camps. On one side were gradient descent methods, such as back-propagation, which produced highly compact models but required thousands of training cycles to converge. On the other side were constructive or memory-based methods, like k-nearest neighbors or Parzen windows. These could learn almost instantly but grew linearly with the size of the training set, leading to bloated models that generalized poorly.
John Platt's research addresses this "missing middle" by introducing the Resource-Allocation Network (RAN). RAN is a hybrid radial-basis-function (RBF) network designed to automatically adjust its complexity to match the target function. By combining the speed of memorization with the efficiency of gradient descent, RAN achieves the compactness of back-propagation with significantly faster training times.
The Dual-Strategy Methodology
RAN operates as a two-layer RBF network that employs a unique dual-strategy learning rule to build its architecture dynamically:
- Memorization (Allocation): When the network encounters a pattern that is both "novel" (far from existing RBF centers) and "difficult" (results in high prediction error), it allocates a new computational unit. The new unit's center is set directly to the input vector, and its weight is initialized to correct the current error. This effectively "memorizes" that specific point in the feature space.
- Gradient Descent (Refinement): If the input is already close to an existing center or the prediction error is low, the network refines its existing knowledge rather than adding a new unit. It uses standard Least Mean Squares (LMS) gradient descent to tweak the weights, centers, and widths of the existing units.
A crucial component of this process is a coarse-to-fine strategy. The distance threshold for adding new units starts large and decays over time. This allows the network to capture the broad, global shape of the function early in training before drilling down into fine-grained, local details.
Experimental Results
The primary benchmark used to evaluate RAN was the Mackey-Glass chaotic time series, a notoriously difficult task for neural networks due to its short-range coherence and long-term unpredictability. The network was trained to predict future values based on four past values, utilizing both off-line (limited data, multiple epochs) and on-line (large data stream, single pass) learning paradigms.
RAN demonstrated remarkable computational efficiency. For the Mackey-Glass series, RAN reached target accuracy in approximately 8 minutes on a SUN-4 workstation. In stark contrast, standard back-propagation required 30 to 60 minutes on a Cray X-MP supercomputer to achieve similar results. As shown in the figure above, RAN achieves a highly favorable trade-off between normalized RMS error and the number of weights, rivaling the compactness of back-propagation while vastly outperforming standard and K-means RBF networks.
When optimized for size, RAN proved significantly more efficient than competing constructive methods like Hashing B-splines. As detailed in the table above, RAN required 2 to 8 times fewer units to achieve comparable or superior accuracy. On the Mackey-Glass task, RAN achieved a normalized RMS error as low as 0.02, matching the performance of much more computationally expensive models.
Key Insights and Limitations
The success of RAN hinges on its two-part allocation condition. A unit is only added if the input is far from existing centers and the error is high. Relying solely on distance would waste resources, while relying solely on error would create unnecessary fine-scale units to fix coarse-scale problems. Consequently, RAN's network growth is sublinear, eventually plateauing once the function is sufficiently mapped.
Despite its strengths, the original RAN architecture has limitations:
- Parameter Sensitivity: Performance depends heavily on hyperparameters like the initial/final distance thresholds, decay constant, and error threshold, without a universal heuristic for setting them.
- Lack of Pruning: While the network stops growing, it lacks a mechanism to remove units that become redundant after gradient descent shifts other centers.
- High-Dimensional Scaling: The paper focuses on chaotic time series and does not fully explore how the distance-based allocation scales to very high-dimensional input spaces, where the "curse of dimensionality" may degrade performance.
Summary updated