Computing with Arrays of Bell-Shaped and Sigmoid Functions

Pierre BaldiNeurIPS 1990 · OpenReview

In the early 1990s, the "connectionist" movement was rapidly advancing the empirical success of neural networks, yet the mathematical foundations explaining why these models worked so well were still being solidified. In this classic paper, Pierre Baldi provides an elegant bridge between biological intuition and rigorous mathematics. By viewing neural network activations through the lens of 18th-century physics and 20th-century probability, Baldi offers a profound theoretical understanding of feed-forward networks.

The Core Problem: Universal Approximation

The paper tackles a fundamental question in computational theory: Can a simple, two-layer neural network approximate any "reasonable" continuous function? While universal approximation is widely accepted today, Baldi sought to provide a more intuitive, constructive proof. He specifically analyzed networks with one hidden layer of "bell-shaped" units (such as Gaussians) or "sigmoid" units, paired with linear output units. Beyond merely proving that these networks can represent functions, Baldi investigated the mathematical "landscape" of learning to understand the difficulty of finding the optimal weights.

Mathematical Foundations: Polynomials and Physics

Baldi employs two primary mathematical frameworks to deconstruct the behavior of these networks:

1. Bernstein Polynomials and Probability

Baldi uses Bernstein polynomials—originally developed to prove the Weierstrass Approximation Theorem—as a mathematical proxy for bell-shaped hidden units. By treating the network's input as the probability of success in a binomial coin-flip experiment, he demonstrates that a linear combination of these "bell curves" naturally converges to the target function as the number of hidden units increases. While Bernstein polynomials are known to converge slowly, they are incredibly "smooth" and preserve the qualitative shape (such as convexity) of the original function. This shape preservation is often more critical for biological systems and robust generalization than raw numerical precision.

2. The Heat Equation and Time Reversal

In one of the paper's most striking insights, Baldi models the output of a Gaussian-based network as the solution to a physical diffusion process. If the weights of the network represent an "initial temperature" distribution, the output of the network corresponds to the "temperature" after a certain amount of time has passed.

Consequently, finding the optimal weights for a network is mathematically equivalent to reversing the heat equation. In physics, heat naturally diffuses and spreads out, leading to a loss of localized information. Learning, therefore, is the attempt to observe this blurred, spread-out information and reconstruct the exact locations and intensities of the original heat sources.

Unifying Sigmoids and Bell-Shaped Units

To bridge the gap between bell-shaped functions (like Radial Basis Functions) and the more ubiquitous sigmoid (S-shaped) activation functions, Baldi introduces a clever geometric mapping. By normalizing the inputs and weights so that they lie on the surface of a high-dimensional sphere, he shows that a sigmoid's response to a dot product effectively creates a localized, bell-shaped "patch" of activity on that sphere.

This spherical mapping reveals that sigmoid units can act like localized detectors (often conceptually linked to "grandmother cells" in neuroscience). This provides a unified heuristic and a shared mathematical foundation proving why both radial and sigmoid networks function as universal approximators.

Optimization and the Generalization Conjecture

Baldi also explores the error landscape of these networks. He notes that if a network only adjusts its linear output weights while keeping the hidden unit parameters fixed, the error landscape is strictly convex. This guarantees the absence of local minima, providing a "straight shot" to the optimal solution via linear regression.

Furthermore, the paper touches upon a conjecture regarding generalization and network capacity. Baldi suggests that as the number of hidden units increases, the learning landscape becomes "smoother" and easier to navigate. This early insight hints at a phenomenon that remains a hot topic in modern artificial intelligence: why highly over-parameterized networks often converge more easily and generalize better than minimal ones.

Limitations and the Curse of Dimensionality

Despite the elegance of these proofs, Baldi acknowledges significant practical challenges:

  • The Ill-Posed Nature of Diffusion: Reversing the heat equation is mathematically "ill-posed" in the sense of Hadamard. This means that microscopic amounts of noise in the target function can lead to massive, unstable swings in the required network weights. While techniques like least-squares regression can mitigate this instability, it remains a fundamental challenge for achieving high-precision learning.
  • The Curse of Dimensionality: While the theoretical framework extends to multi-dimensional spaces, the number of bell-shaped units required to adequately "cover" the input space grows exponentially as dimensionality increases.

Conclusion

By connecting neural network architecture to Bernstein polynomials and the heat equation, Baldi's work provides a deeply constructive proof of universal approximation. It elegantly reframes the learning process as the reversal of physical diffusion, offering timeless insights into the optimization, capacity, and fundamental nature of artificial neural networks.

Summary updated

Abstract

We consider feed-forward neural networks with one non-linear hidden layer and linear output units. The transfer function in the hidden layer are ei(cid:173) ther bell-shaped or sigmoid. In the bell-shaped case, we show how Bern(cid:173) stein polynomials on one hand and the theory of the heat equation on the other are relevant for understanding the properties of the corresponding networks. In particular, these techniques yield simple proofs of universal approximation properties, i.e. of the fact that any reasonable function can be approximated to any degree of precision by a linear combination of bell(cid:173) shaped functions. In addition, in this framework the problem of learning is equivalent to the problem of reversing the time course of a diffusion pro(cid:173) cess. The results obtained in the bell-shaped case can then be applied to the case of sigmoid transfer functions in the hidden layer, yielding similar universality results. A conjecture related to the problem of generalization is briefly examined.
BibTeX
@inproceedings{baldi1990computing,
  author = {Pierre Baldi},
  title = {{Computing with Arrays of Bell-Shaped and Sigmoid Functions}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1990},
  url = {https://openreview.net/forum?id=ByEeKU-ubH},
}
Figure