Geometric Expressivity Limits of Shallow ReLU Networks

15 concepts8 papers1 proposal

The Universal Approximation Theorem is a foundational comfort in deep learning: it guarantees that a feedforward neural network with a single hidden layer can approximate any continuous function to arbitrary precision, provided it is given sufficient width. However, this guarantee hides a crucial distinction between approximation and exact representation.

While a shallow network can get arbitrarily close to a target function within a bounded domain, it often faces absolute geometric boundaries that prevent exact representation. When we look closely at the mathematical structure of shallow architectures—particularly those utilizing Rectified Linear Units (ReLU)—we find strict constraints imposed by the geometry of linear neuron combinations.

The Gap Between Approximation and Representation

To understand why shallow networks face hard representational limits, we must look at their fundamental building blocks. A standard artificial neuron computes a linear weighted sum of its inputs followed by a simple activation function. As noted by Nadav Cohen et al. in their work on Deep SimNets, and reiterated by Abel A. Reyes et al. in the context of NExNet Seg, this reliance on linear inner products inherently limits the expressive power of individual units, forcing the broader architecture to rely heavily on depth or width to construct complex non-linear mappings.

When these linear neurons are combined with ReLU activations, the network's output becomes a piecewise linear (or piecewise affine) function. The input space is carved up into a collection of distinct polyhedral regions, inside each of which the network behaves as a simple linear function. The boundaries between these regions are called "breaklines" or "creases."

A common misconception is that because shallow ReLU networks produce piecewise linear outputs, they can represent any continuous piecewise linear function. As Li Li 0013 et al. demonstrate in Piecewise linear neural networks and deep learning, this is fundamentally untrue in higher dimensions. There is a rigid structural gap between the space of all piecewise linear functions and the subset that a single hidden layer can actually construct.

The Hard Geometry of Creases and Intersections

Why does this gap exist? The intuition lies in how a shallow network constructs its creases.

Every hidden neuron in a single-layer ReLU network introduces a single, infinitely extending flat crease (a hyperplane) into the input space. The final output of the shallow network is simply a weighted sum of these individual, global creases.

Shallow Network (Summation):
Neuron 1: ─────────────────────────── (Global Crease)
Neuron 2: \ \ \ \ \ \ \ \ \ \ \ \ \ \ (Global Crease)
Result:   A simple overlay of global lines. Creases cannot stop or bend abruptly.

Because the network only performs a linear combination (a sum) of these units, it cannot easily isolate or terminate a crease. If a target piecewise linear function contains creases that intersect and stop abruptly at a specific vertex, a shallow network attempting to match this intersection will inadvertently extend those creases across the entire input space.

Steffen Dereich et al. formalize this constraint in On minimal representations of shallow ReLU networks. The authors introduce a geometric representability test based on the dimensionality of hyperplane intersections. They prove that if too many breaklines intersect at a single point in a specific geometric configuration, the function becomes completely unrepresentable by a shallow ReLU network—regardless of how many neurons are added to that single layer. The local arrangement of the linear regions creates a geometric contradiction that a flat summation cannot resolve.

Curved Boundaries and the Infinite Norm Penalty

The representational bottleneck worsens when moving from sharp, linear intersections to functions with curved or intricate boundaries.

Consider the "singular set" of a function—the set of points where the function is non-smooth or non-differentiable. For a ReLU network, this singular set is always a collection of flat hyperplanes. If we try to represent a function whose singular set is smoothly curved (such as the distance function from a smooth sphere) or fractal, a shallow network faces an impossible task.

In Representation formulas and pointwise properties for Barron functions, Weinan E et al. analyze infinitely wide two-layer networks within the "Barron space"—a framework where networks have infinite width but are constrained to a finite path-norm (meaning their weights remain bounded). The authors reveal that such networks are inherently incapable of representing functions with complex or curved singular sets. To perfectly match a curved boundary, the network would need to arrange infinitely many flat hyperplanes to smooth out the curve, which causes the required network norm to explode to infinity.

A similar phenomenon occurs when we remove the safety net of a bounded input domain. In A Gap Between the Gaussian RKHS and Neural Networks: An Infinite-Center Asymptotic Analysis, Akash Kumar 0010 et al. explore the entire Euclidean space $\mathbb{R}^d$. They identify specific, smooth functions belonging to the Gaussian Reproducing Kernel Hilbert Space (RKHS) that a simple Gaussian kernel handles easily, but which a shallow ReLU network cannot represent at all. Because the domain is unbounded, trying to mimic the localized, smooth decay of a Gaussian function using flat, linearly growing ReLUs forces the network's required norm to become infinite.

Beyond the Linear Neuron Bottleneck

Understanding these rigid boundaries explains why modern deep learning is so dependent on alternative structural dimensions like depth, equilibrium states, and architectural "height."

To bypass the geometric limitations of a single layer, traditional feedforward networks stack layers sequentially. This allows the network to compose functions rather than just sum them. Composition allows the network to bend, crop, and localize creases, creating highly complex polyhedral regions with far fewer parameters. As Zhoutong Wu et al. demonstrate in Separation and Bias of Deep Equilibrium Models on Expressivity and Learning Dynamics, standard feedforward architectures require massive depth to efficiently represent intricate piecewise linear structures, whereas implicit models like Deep Equilibrium Models (DEQs) can capture them through iterative fixed-point dynamics.

Alternatively, researchers have looked at expanding the geometry of the layers themselves. Fenglei Fan et al. tackle this in On Expressivity of Height in Neural Networks by introducing intra-layer connections, creating a "3D" network topology.

Deep Composition (Layering):
Layer 1: ───┐  ┌───  (Bends the crease)
            └──┘
Layer 2: Composes the bent shapes to isolate a local region.

By allowing neurons within the same layer to talk to one another, the network can generate a massive number of linear "pieces" and resolve complex hyperplane intersections locally. This architectural modification breaks the rigid geometric constraints of standard shallow layers, providing a more direct path to high expressivity without requiring extreme depth.

Key Foundational Work

Representation formulas and pointwise properties for Barron functions (2020) — contributed 4 concept elements to this direction.
Deep SimNets (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_singular_set_limitations_d93ad1ee,
  title = {singular set limitations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/singular-set-limitations/rdr_d93ad1ee13dd41eabe87eb501daccf74},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_singular_set_limitations_d93ad1ee_v1,
  title = {singular set limitations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/singular-set-limitations/rdr_d93ad1ee13dd41eabe87eb501daccf74/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake