Splines, Rational Functions and Neural Networks

Robert C. Williamson, Peter L. BartlettNeurIPS 1991 · OpenReview

Feedforward neural networks are ubiquitous in nonlinear regression, yet their fundamental approximation-theoretic properties—how efficiently and accurately they can represent a target function—were historically underexplored. This paper bridges the gap between neural networks and classical approximation theory by drawing rigorous mathematical comparisons to splines and rational functions. Ultimately, the authors demonstrate that for certain smooth functions, adding hidden layers does not necessarily improve the theoretical "degree of approximation." Furthermore, they provide new error bounds for splines when knots are chosen based on data distribution rather than the function itself.

The Core Problem

The authors tackle two primary theoretical questions:

  1. The Multi-Layer Advantage: Does a two-hidden-layer network inherently provide a better degree of approximation than a single-layer network, or does it merely add complexity without a theoretical boost in accuracy?
  2. Data-Dependent Approximation: In statistical learning, the target function is rarely known perfectly; instead, we rely on samples. How does approximation error behave if the "knots" (points where a piecewise function changes) are placed based on the probability distribution of the data rather than the shape of the target function?

Theoretical Framework

Rather than relying on empirical experiments or specific datasets, this work is grounded in heavy-duty mathematical and functional analysis. The authors leverage:

  • Vitushkin’s Theory: Applying results from "piecewise rational" (PR) functions to establish lower bounds of approximation. Because neural networks with certain activation functions behave like compositions of rational functions, this serves as a proxy for studying multi-layer architectures.
  • Besov and Sobolev Spaces: These mathematical frameworks measure the "smoothness" of functions. The authors extend existing results from Sobolev spaces (which focus on derivatives) to more general Besov spaces.
  • Spline Theory: By utilizing the known relationship between splines with free knots and rational functions, the authors draw direct parallels to neural network architectures.

Key Insights and Results

The Diminishing Returns of Layers

One of the most striking insights is that for single-variable smooth functions (within Sobolev spaces), using multiple hidden layers (represented as compositions of rational functions) does not yield a significantly better degree of approximation than a single layer. Theorem 3.1 establishes lower bounds showing that the improvement gained by "layering" these functions is marginal for the studied function classes. This suggests that the empirical success of deep layers may stem more from optimization dynamics or multivariable complexity than from basic approximation rates for simple smooth functions.

Neural Networks as High-Dimensional Splines

The paper highlights that a single-hidden-layer network using a piecewise linear "hinge" activation function is essentially a canonical high-dimensional spline.

Knot Placement and Data Distribution

Theorem 4.2 provides a concrete error bound for spline approximation when knots are chosen according to a probability measure $\mu$ rather than the target function $f$. The error is bounded by $cn^{-\alpha} \|f\|$, where $n$ is the number of knots. This is crucial for real-world machine learning, where the data distribution is often the only observable element.

Non-Uniform Distribution Penalty

The findings indicate that the degree of approximation is generally worse under non-uniform data distributions compared to uniform ones. The departure from a uniform distribution effectively acts as a penalty term in the error bound.

Limitations and Future Directions

  • The Curse of Dimensionality: Most definitive results in this paper apply to single-variable functions. Extending these theoretical bounds to multivariable approximation remains an open problem and an obvious next step.
  • Achievability of Bounds: While the paper provides theoretical lower bounds (the absolute best performance limit), it remains unknown whether the specific bound for multi-layer rational compositions is practically achievable.
  • Shifting Focus to Simpler Models: The authors suggest a provocative conclusion: if neural networks are theoretically no better than "dyadic splines" (a specific, simple type of spline) in these contexts, researchers might benefit from shifting their focus toward the learning theory (such as PAC learning) of those simpler spline representations.

Summary updated

Abstract

Connections between spline approximation, approximation with rational functions, and feedforward neural networks are studied. The potential improvement in the degree of approximation in going from single to two hidden layer networks is examined. Some results of Birman and Solomjak regarding the degree of approximation achievable when knot positions are chosen on the basis of the probability distribution of examples rather than the function values are extended.
BibTeX
@inproceedings{williamson1991splines,
  author = {Robert C. Williamson and Peter L. Bartlett},
  title = {{Splines, Rational Functions and Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1991},
  url = {https://openreview.net/forum?id=B1WUfvWOWr},
}
Figure