Synergy of Clustering Multiple Back Propagation Networks

William P. Lincoln, Josef SkrzypekNeurIPS 1989

Introduction

This paper demonstrates that a "cluster" of multiple small back-propagation (BP) networks can significantly outperform a single, larger network in both generalization and fault tolerance. By averaging the outputs of several independently initialized networks and using that consensus as a feedback signal for self-training, the system achieves a synergistic effect. Essentially, the collective "wisdom" of the group corrects the individual failures of its members.

The Problem with Single Networks

In the late 1980s, the limitations of standard back-propagation were becoming increasingly clear. Networks often failed to learn complex mappings completely or generalized poorly to new, unseen data. The standard "brute force" solution at the time—increasing the number of hidden layers or neurons—frequently led to diminishing returns. In some cases, it even worsened performance due to credit assignment difficulties.

To overcome these hurdles, the authors investigate a modular approach: clustering multiple simple networks. They specifically address how to maintain performance when a "perfect teacher" (labeled data) is unavailable, and how to make neural systems resilient to hardware failures, such as broken connections or links.

The Modular Solution: Clustering

The researchers implemented a cluster of five three-layered back-propagation networks. Each network was initialized with different starting weights, ensuring they developed unique internal representations of the same task.

Architecture of the neural network cluster, showing multiple independent networks feeding into a central Judge.

The core of the system is the "Judge," which combines the outputs of the individual networks into a single cluster output using two primary rules:

  1. Simple Average: A straight mean of all network outputs.
  2. Convex Combination: A weighted average where the Judge biases the output toward networks that have historically been more reliable.

The Feedback Loop: A key innovation of this architecture is using the cluster's consensus output as the "desired output" to retrain individual networks when a human teacher is absent. This allows the cluster to self-organize and adapt to changing conditions or internal faults.

Experimental Setup

The authors used a synthetic, abstract learning problem to test their theories:

  • The Task: Categorize 3-dimensional points $P = (x, y, z)$ into one of eight sets.
  • The Geometry: The sets are defined as eight spheres of radius 1, centered at $(\pm1, \pm1, \pm1)$.
  • Network Architecture: 3 input nodes (coordinates), one hidden layer (typically 15 units), and 8 output nodes (one for each sphere).
  • Generalization Test: The networks were evaluated using points located outside the original training spheres.

Key Insights

  • The "Fooling" Maxim: The authors summarize the cluster advantage with a play on Abraham Lincoln's famous quote: "You can fool some of the single BP's in a cluster all of the time, but you cannot fool all of them all of the time." Because each net starts with different weights, they rarely make the same mistakes simultaneously.
  • Non-Linear Scaling: The paper argues against the "more nodes is better" myth. They show that a single network's error is a non-linear function of its hidden units; adding more nodes can actually make learning harder by diluting credit assignment.
  • Self-Correction via Consensus: By feeding the cluster's average back to the individuals, the system can "heal" a damaged network. If one network suffers a link failure, the other four "vote" to pull the damaged network back toward the correct mapping.
  • The Drift Trade-off: While retraining on the cluster's own output adds fault tolerance, it introduces the risk of "drift"—where the entire system slowly wanders away from the true mapping if the initial training wasn't sufficiently accurate.

Key Results

Cluster Advantage: The authors measured the ratio of individual net error to cluster error. They found that the cluster advantage is most significant when individual networks have learned the task "marginally." If the networks are already perfect or haven't learned at all, the collective advantage disappears.

Graph demonstrating that the cluster advantage peaks when individual networks have a marginal error rate.

Fault Tolerance: In simulations where links were forced to fail (weights set to 0) at a rate of 1 link per 350 inputs, the cluster maintained significantly lower error rates than a single network.

Learning in the Absence of a Teacher: The cluster was able to continue reducing its error even after the teacher was removed, provided it used the Judge’s output as a training signal.

Limitations and Future Work

  • Task Complexity: The experiments were limited to a relatively simple 3D geometric classification task. The authors acknowledge that while the results seem general, they need to be tested on much larger, more complex systems.
  • The Drift Problem: The paper notes that if the initial training is poor, the feedback loop can amplify errors rather than correct them. Finding the "stability point" where feedback becomes consistently beneficial remains a challenge.
  • Weight Adjustment Rules: While they proposed a convex combination rule for the Judge, the paper primarily focuses on the simple average. More sophisticated "judging" algorithms could potentially yield even higher synergy.

Summary updated

BibTeX
@inproceedings{lincoln1989synergy,
  author = {William P. Lincoln and Josef Skrzypek},
  title = {{Synergy of Clustering Multiple Back Propagation Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1989},
}
Figure