Relaxation Networks for Large Supervised Learning Problems

Joshua Alspector, Robert B. Allen, Anthony Jayakumar, Torsten Zeppenfeld, Ronny MeirNeurIPS 1990 · OpenReview

TL;DR

This paper demonstrates that relaxation-based neural networks—specifically Boltzmann Machines and Mean-Field Theory (MFT) models—possess the same representational power as back-propagation, despite being computationally slower on serial computers. To overcome this speed bottleneck, the authors present a custom VLSI microchip that implements these "hardware-friendly" local learning rules, promising parallel learning speeds far exceeding traditional software simulations.

The Challenge with Back-Propagation in Hardware

In the early 1990s, back-propagation (BP) was the dominant algorithm for supervised learning due to its efficiency on serial processors. However, BP is notoriously difficult to implement directly in hardware (VLSI) because it requires a global distribution of error signals.

In contrast, "relaxation networks" like the Boltzmann Machine use feedback connections and local learning rules, which are much easier to cast into silicon. The skepticism at the time was whether these networks could actually scale to "usefully large" problems or if they were merely theoretical curiosities. The authors tackle two main questions:

  1. Do these networks scale as well as back-propagation?
  2. Can we build a physical system that makes their learning speeds competitive?

Methods and Hardware Co-Design

The authors compare three distinct approaches:

  • Back-propagation (BP): The standard gradient descent method.
  • Boltzmann Machine (BM): A stochastic network that uses "noise" (simulated annealing) to find stable states.
  • Mean-Field Theory (MFT): A deterministic version of the Boltzmann Machine. Instead of using random noise, it uses a "gain sharpening" technique (gradually increasing the slope of the activation function) to approximate the stochastic process more efficiently.

The Learning Rule: Both BM and MFT use a two-phase learning process. In the "plus" phase, outputs are clamped to the target; in the "minus" phase, they run free. The weights are adjusted based on the difference in local correlations between these phases. Crucially, the authors used a sgn (sign) function for weight updates to match the constraints of their digital/analog hybrid hardware.

VLSI Microchip

The authors didn't just write code; they designed the algorithm to fit the chip. For instance, they used 5-bit weight resolution because that was what their VLSI process could reliably store, proving that high-precision floating-point numbers aren't always necessary for complex learning.

Datasets

The study evaluated the networks on three benchmarks:

  1. Parity Problem: A classic "hard" problem (generalization of XOR) where the output depends on whether the number of bits is even or odd.
  2. Replication (Identity) Problem: A task where the network must duplicate the input pattern at the output layer after passing it through a hidden layer.
  3. NETtalk: A large-scale, real-world corpus of 20,000 words used to train networks to map English text to phonemes (speech sounds).

Key Results

The paper reinforces that global optimization can emerge from purely local synaptic updates—the "holy grail" for neuromorphic hardware. Mean-Field Theory provides a deterministic "shortcut" to the Boltzmann Machine, running roughly 10x faster than stochastic Boltzmann learning in simulation while maintaining similar accuracy.

Scaling Comparison
  • Scaling Parity and Replication: For the parity and replication problems, there was no significant difference in the number of patterns required for BP, BM, or MFT to learn. All three scaled exponentially with input size for parity.
  • NETtalk Performance: On the complex NETtalk task, back-propagation performed slightly better than the relaxation networks. The authors attribute this to BP's superior ability to handle the ambiguity of mapping single letters to multiple possible phonemes.
  • Hardware Speedup: The fabricated 32-neuron chip (featuring 496 bidirectional synapses) demonstrated that it could potentially process 100,000 patterns per second. While MFT is 10x slower than BP on a standard computer, the parallel hardware implementation would make it orders of magnitude faster than any BP simulation.

Limitations and Future Work

  • Simulation Bottleneck: On serial hardware, relaxation networks are painfully slow (BM is 100x slower than BP). This makes research difficult without the specialized chips the authors are developing.
  • Network Size: The current chip is limited to 32 neurons. While the authors designed the chips to be "cascadable" (linked together to form larger networks), the paper only shows results for a single-chip scale or software simulations of larger ones.
  • Weight Decay: For the largest problems, the authors found they needed to add a "global decay" to the weights to maintain learning stability, suggesting that managing dynamic range in hardware remains a challenge.

Summary updated

Abstract

Feedback connections are required so that the teacher signal on the output neurons can modify weights during supervised learning. Relaxation methods are needed for learning static patterns with full-time feedback connections. Feedback network learning techniques have not achieved wide popularity because of the still greater computational efficiency of back-propagation. We show by simulation that relaxation networks of the kind we are implementing in VLSI are capable of learning large problems just like back-propagation networks. A microchip incorporates deterministic mean-field theory learning as well as stochastic Boltzmann learning. A multiple-chip electronic system implementing these networks will make high-speed parallel learning in them feasible in the future.
BibTeX
@inproceedings{alspector1990relaxation,
  author = {Joshua Alspector and Robert B. Allen and Anthony Jayakumar and Torsten Zeppenfeld and Ronny Meir},
  title = {{Relaxation Networks for Large Supervised Learning Problems}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1990},
  url = {https://openreview.net/forum?id=SJZ4zPZObH},
}
Figure