Metamorphosis Networks: An Alternative to Constructive Models

Brian V. Bonnlander, Michael Mozer

The Challenge: Finding the "Goldilocks" Zone of Complexity

In machine learning, determining the appropriate number of free parameters for a model is a notoriously difficult challenge, often referred to as the bias/variance dilemma. If a network has too few parameters, it cannot capture the underlying patterns in the data (high bias). If it has too many, it overfits to the noise in the training data (high variance).

Traditional "constructive" algorithms, such as the Resource-Allocating Network (RAN) or Cascade-Correlation, attempt to solve this by starting with a small network and adding new hidden units as learning progresses. However, these methods have notable flaws. They can be overly sensitive to noise, frequently adding units based on individual outliers. Furthermore, in high-dimensional spaces, they often add far more parameters than are actually necessary to solve the problem.

To address these shortcomings, researchers have proposed an alternative class of algorithms called Metamorphosis Networks. Instead of adding new neurons to handle complex data, these networks maintain a fixed number of units but gradually relax constraints on their parameters. This effectively "unlocks" degrees of freedom only where they are needed to balance the bias/variance tradeoff.

The Metamorphosis Approach

The authors introduce Metamorphosis RBF (MRBF) Networks, which modify standard Radial Basis Function (RBF) architectures through a novel constraint and relaxation process.

1. The Lattice Constraint and Parameter Interpolation

Unlike standard RBF networks where every unit's parameters (center, width, and height) are completely independent, MRBF units are arranged on a logical lattice. The actual parameters of each RBF unit are derived by interpolating between a smaller set of "underlying psets" (parameter sets).

1D lattice constraint showing RBF units interpolated from underlying parameter sets.

Initially, very few underlying psets exist. This forces many RBF units to share similar properties, acting as a heavy initial constraint that promotes high bias and low variance. The network is forced to find a smooth, global approximation of the data first.

2. The Metamorphosis Process

When the network reaches a performance plateau and can no longer reduce error with its current constraints, it undergoes "metamorphosis." The algorithm identifies a region on the lattice exhibiting high "tension"—a state where the error gradients are pulling the parameters of neighboring units in conflicting directions.

Tension on a 2D lattice leading to a region split.

Once a high-tension region is identified, the network splits that region by adding new underlying psets. This recursive partitioning is similar to statistical methods like CART or MARS, but applied directly to the internal parameter space of a neural network. It allows the model to increase its flexibility locally without adding more neurons to the overall architecture.

Key Insights and Advantages

The MRBF architecture provides several distinct advantages over traditional constructive models:

  • Robustness to Noise (Global vs. Local Complexity): By using a "tension" heuristic based on gradients across an entire region, the model is much more robust to noise than algorithms that add units based on a single high-error data point.
  • Variable Subset Selection: MRBFs can effectively ignore irrelevant input dimensions. Because the lattice splitting is strategic, the model only adds degrees of freedom along dimensions that actively help reduce error.
  • Multiresolution Learning: The model naturally learns in a "coarse-to-fine" manner. It first establishes a global, smooth approximation of the data and only introduces sharp, local details (via metamorphosis) when the smooth version is exhausted.
  • Built-in Interpolation: Because the units are constrained by a lattice, the network naturally "fills in the gaps" between sparse data points far better than standard RBF networks, which tend to produce "bumpy" responses in empty spaces.

Experimental Validation

The authors evaluated MRBF networks across several benchmarks, demonstrating significant efficiency and regularization benefits:

  • Mackey-Glass Chaotic Time Series: On this standard non-linear prediction task, MRBF networks achieved comparable accuracy to the popular RAN algorithm but utilized significantly fewer free parameters.
  • Noisy Sine Wave: When tested on a sine wave with varying levels of Gaussian noise, the RAN algorithm's complexity exploded as it attempted to fit the noise. In contrast, the MRBF network's parameter count remained low and stable, demonstrating superior regularization.
  • U-Shape Classification & Sparse Data: The model successfully demonstrated its multiresolution processing capabilities on a 2D toy problem and showed strong qualitative interpolation in regions where training data was missing.

Limitations and Future Directions

The authors note that while MRBF networks offer a highly efficient and robust alternative to constructive models, they do not necessarily yield astonishing improvements in raw generalization error across all tasks compared to heavily tuned existing methods.

Future research directions include exploring different basis functions (such as finite-element methods) instead of RBFs, implementing "soft" constraints using regularization techniques rather than hard lattice interpolation, and investigating supervised learning versions of Kohonen-style updates where modifying one unit's weights automatically influences its neighbors.

Summary updated

Abstract

Given a set oft raining examples, determining the appropriate num(cid:173) ber of free parameters is a challenging problem. Constructive learning algorithms attempt to solve this problem automatically by adding hidden units, and therefore free parameters, during learn(cid:173) ing. We explore an alternative class of algorithms-called meta(cid:173) morphosis algorithms-in which the number of units is fixed, but the number of free parameters gradually increases during learning. The architecture we investigate is composed of RBF units on a lat(cid:173) tice, which imposes flexible constraints on the parameters of the network. Virtues of this approach include variable subset selec(cid:173) tion, robust parameter selection, multiresolution processing, and interpolation of sparse training data.
BibTeX
@inproceedings{bonnlander1992metamorphosis,
  author = {Brian V. Bonnlander and Michael Mozer},
  title = {{Metamorphosis Networks: An Alternative to Constructive Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1992},
  url = {https://openreview.net/forum?id=HJ-RK8-dbB},
}
Figure