Minimax Optimal Convergence of Shallow Neural Networks

30 concepts18 papers

A foundational tenet of deep learning folklore is that network depth is essential. Deep architectures are widely believed to be uniquely capable of breaking the curse of dimensionality, allowing them to represent complex, smooth functions with exponentially fewer parameters than their shallow counterparts.

However, a growing body of theoretical research challenges this assumption. By analyzing neural networks through the lenses of functional analysis, nonparametric statistics, and approximation theory, researchers have shown that shallow networks—often containing only one or two hidden layers—can achieve minimax optimal convergence rates. This means they can learn smooth functions from noisy data with the absolute maximum statistical efficiency theoretically possible for any estimator, proving that depth is not always a prerequisite for optimality.

Matching Activation Regularity to Function Smoothness

Why do shallow networks perform so surprisingly well? The key lies in the interplay between the activation function and the smoothness of the target function. Traditional proofs often relied on standard Rectified Linear Units (ReLU), which are continuous but not continuously differentiable. When approximating highly smooth functions (such as those in Sobolev spaces), standard ReLU networks require significant depth to emulate higher-order smoothness through hierarchical composition.

However, if we broaden the choice of activation functions, the necessity of depth diminishes. Ilsang Ohn et al. established a unified framework in Smooth function approximation by deep neural networks with general activation functions showing that a wide array of smooth activations—including Sigmoid, Tanh, Swish, and ELU—allow networks to achieve minimax optimal convergence rates in regression and classification tasks. Because the network's output inherits the smoothness of its activation function, a single layer can approximate the target function efficiently without needing depth to build up smoothness via composition.

This intuition is further supported by specific architectural choices:

From Parameter Counting to Function Spaces

In classical approximation theory, efficiency is measured by the number of hidden neurons or parameters. Under this metric, shallow networks often appear disadvantaged. The modern breakthrough shifts the focus from parameter counts to norm regularization within specific function spaces.

When a shallow network is heavily over-parameterized (i.e., it has an arbitrarily large number of neurons), its capacity is controlled not by the size of the architecture, but by the magnitude of its weights. Yunfei Yang demonstrated in Nonparametric regression using over-parameterized shallow ReLU neural networks that over-parameterized shallow ReLU networks achieve minimax optimal convergence rates for nonparametric regression, provided their weights are properly constrained via regularization. This norm-based constraint prevents the network from overfitting wildly, allowing an infinitely wide network to behave like a well-conditioned, smooth function space.

To formalize this behavior, researchers map neural networks to "native" Banach spaces:

  1. Representer Theorems: Michael Unser et al. proved in Function-Space Optimality of Neural Architectures with Multivariate Nonlinearities that the exact optimal solutions for data-fitting problems within these native Banach spaces are precisely shallow neural network architectures with multivariate nonlinearities.
  2. Weighted Variation Spaces: To expand the types of functions these networks can efficiently represent, Ronald DeVore et al. introduced "weighted variation spaces" in Weighted variation spaces and approximation by shallow ReLU networks. These intrinsic function spaces are larger than traditional ones but maintain the exact same optimal approximation rates, providing a precise mathematical foundation for why shallow ReLU networks are highly expressive in high-dimensional settings, a property also explored by Tong Mao et al. in Rates of approximation by ReLU shallow neural networks.

Mechanisms of Optimality: Greedy Selection and Error Correction

Beyond static representation, how do these shallow architectures dynamically build optimal approximations? Two primary mechanisms explain this efficiency: dictionary-based greedy selection and iterative error correction.

A shallow network can be mathematically framed as a linear combination of atoms chosen from a dictionary, where each atom is a single neuron with a specific weight and bias. Jonathan W. Siegel et al. showed in Optimal Convergence Rates for the Orthogonal Greedy Algorithm that the dictionary properties of shallow Heaviside and ReLU networks enable the Orthogonal Greedy Algorithm (OGA) to achieve accelerated convergence rates that surpass classical $O(n^{-1/2})$ bounds. This directly links iterative optimization to optimal approximation guarantees.

Alternatively, shallow networks can achieve high expressivity through a "corrective mechanism." Guy Bresler et al. proved in A Corrective View of Neural Networks: Representation, Memorization and Learning that instead of relying on deep hierarchical features, wide shallow networks can be organized into groups of neurons where subsequent groups are trained to iteratively correct the residual errors of preceding groups. This architectural mechanism allows shallow networks to represent highly smooth functions and learn low-degree polynomials with sub-polynomial complexity, bypassing the need for depth entirely.

[Input Data] 
     │
     ├──► [Neuron Group 1] ──► Initial Approximation (with residual error)
     │                                │
     └──► [Neuron Group 2] ───────────┴──► Corrective Step (erases initial error)
                                              │
                                              ▼
                                    [Optimal Smooth Output]

Aggregation and Functional Priors

Finally, the optimality of shallow networks extends to probabilistic and ensemble settings. Instead of searching for a single optimal network configuration, combining multiple shallow networks can yield optimal estimators. Laura Tinsi et al. proved in Risk bounds for aggregated shallow neural networks using Gaussian priors that aggregated shallow networks using sigmoid activations achieve minimax-optimal convergence rates for Sobolev smoothness classes, guaranteeing that no other statistical estimator can fundamentally outperform an ensemble of shallow models.

This principle also applies to Bayesian neural networks (BNNs). Marcin Sendera et al. demonstrated in Hi-fi functional priors by learning activations that a single wide hidden layer in a BNN can approximate complex functional priors—matching the behavior of deep architectures or Gaussian processes—simply by making the activation functions learnable. This significantly reduces the computational overhead of deep models while retaining their expressive power.

While deep networks remain highly effective in practice, this collective body of work reveals that depth is not a strict mathematical prerequisite for optimal learning. Through proper activation function selection, norm-based regularization, and greedy or corrective optimization mechanisms, shallow architectures achieve the theoretical limits of minimax optimal convergence, redefining our understanding of efficiency in deep learning theory.

Key Foundational Work

A Corrective View of Neural Networks: Representation, Memorization and Learning (2020) — contributed 2 concept elements to this direction.
Approximation rates for neural networks with general activation functions (2020) — contributed 1 concept elements to this direction.
Smooth function approximation by deep neural networks with general activation functions (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_minimax_optimal_convergence_61ff51f3,
  title = {minimax optimal convergence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/minimax-optimal-convergence/rdr_61ff51f3554446b1b84d64adc33be4fa},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_minimax_optimal_convergence_61ff51f3_v1,
  title = {minimax optimal convergence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/minimax-optimal-convergence/rdr_61ff51f3554446b1b84d64adc33be4fa/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake