Universal Approximation under Architectural Constraints

21 concepts15 papers1 proposal

The classic Universal Approximation Theorem provides a comforting guarantee: a neural network with a single, arbitrarily wide hidden layer can approximate any continuous function to any desired precision. However, modern deep learning rarely uses unconstrained, infinitely wide layers. Instead, practical architectures are defined by strict structural constraints designed for specific computational or physical priors.

Convolutional Neural Networks (CNNs) enforce local connectivity and weight sharing. Invertible Neural Networks (INNs) mandate strict bijectivity for exact density estimation. Other models restrict network width to save memory or enforce exact boundary conditions to satisfy physical laws.

Do these structural constraints break the fundamental expressivity of neural networks? A growing body of research explores this tension, proving that constrained networks can retain their universal approximation capabilities by leveraging deep geometric and topological principles.

The Geometry of Invertibility and Diffeomorphisms

Invertible Neural Networks (INNs) are widely used in generative modeling and normalizing flows because they allow for exact likelihood computation and bidirectional mapping. However, ensuring invertibility requires highly restricted layer designs. For example, affine coupling layers (used in models like RealNVP) split the input coordinates into two halves, keeping one half fixed while using it to apply a simple linear shift and scale to the other half.

Intuitively, this coordinate-wise restriction feels too limiting. If a network can only alter a subset of dimensions at a time, can it still deform space into any arbitrary shape?

To answer this, researchers reframe universal approximation through the lens of differential geometry. Instead of approximating arbitrary continuous functions, an INN aims to approximate diffeomorphisms—smooth, invertible mappings between spaces.

Unconstrained Network: Can tear, glue, or collapse space.
Invertible Network (Diffeomorphism): Smoothly stretches and twists space like a sheet of rubber, preserving topology.

In a foundational proof, Ishikawa et al. (2020) demonstrated that affine coupling layers are theoretically sufficient to approximate any smooth diffeomorphism. By stacking these simple, coordinate-wise transformations and alternating which dimensions are frozen, the network can compose complex global deformations. Ishikawa et al. (2022) extended this guarantee to other restricted architectures, including Neural Ordinary Differential Equations (Neural ODEs), confirming that the mathematical constraints required for bijectivity do not inherently limit a model's representational capacity.

Furthermore, this diffeomorphic capability can be used to approximate non-invertible functions. Hwang (2025) showed that any complex, continuously differentiable function can be decomposed into a combination of an affine transformation and a diffeomorphism. By focusing the network's capacity on representing the smooth, invertible component, even highly constrained architectures can serve as universal approximators for broader classes of smooth functions.

The Expressivity Gap in Practice

While constrained networks are theoretically universal approximators, satisfying these constraints in practice often introduces an "expressivity gap" or optimization bottleneck.

For instance, Chan et al. (2023) demonstrated that bijective networks constrained to a fixed width equal to the input dimension face severe theoretical limitations. While increasing network depth can theoretically mitigate this bottleneck, these fixed-width models struggle to represent complex distributions compared to architectures that allow for intermediate dimension expansion. Similarly, both Xie et al. (2021) and Kwon et al. (2021) observed that the strict mathematical requirements of invertibility make it difficult for INNs to capture the highly non-linear features necessary for high-fidelity image restoration and compression, often necessitating hybrid designs with non-invertible paths.

To inject expressivity back into constrained models without sacrificing their mathematical guarantees, researchers have focused on designing more flexible, learnable activation functions:

  • Trainable Diffeomorphic Activations: Chelly et al. (2024) introduced DiTAC, a class of trainable activation functions derived from one-dimensional velocity fields. These functions create smooth, monotonically increasing mappings that remain strictly differentiable and invertible while significantly boosting network flexibility.
  • Graph-Adaptive Transformations: For non-Euclidean domains, Mantri et al. (2024) developed DiGRAF, which warps a neuron's output space using continuous piecewise-affine mappings that adapt to the input graph's structure. Because these transformations are diffeomorphic, they ensure global differentiability, avoiding the optimization instabilities common to non-smooth adaptive activations.

Even when constraints are pushed to extremes—such as requiring the network to be entirely volume-preserving (measure-preserving)—universal approximation is still achievable. Zhu et al. (2021) proved that measure-preserving architectures (like NICE or RevNets) can approximate any bounded, injective, measure-preserving map to arbitrary precision, validating their use in memory-efficient training and generative modeling.

Locality, Sparsity, and Width Constraints

The tension between constraint and expressivity extends beyond invertibility to fundamental architectural choices like weight sharing, sparsity, and width.

Consider a standard 2D Convolutional Neural Network (CNN) with small $3 \times 3$ filters. A fully connected layer can look at an entire input vector simultaneously, whereas a convolutional layer is forced to look only at local neighborhoods and apply the exact same weights across the entire input space. Intuitively, this extreme sparsity and weight sharing should limit the network's capacity.

However, depth acts as a powerful equalizer. Zhou (2018) provided a rigorous proof that deep CNNs are universal approximators, demonstrating that as depth increases, the composition of local operations allows the network to represent any continuous function while using significantly fewer parameters than a fully connected counterpart.

Going a step further, Oono et al. (2019) and He et al. (2022) proved that deep, constant-width residual CNNs can achieve the exact same asymptotic "minimax optimal" error rates as fully connected networks for smooth function classes. This confirms that the structural constraints of convolution and residual connections do not diminish a network's fundamental capacity to learn complex, high-dimensional distributions.

Fully Connected Layer: High parameter count, global view instantly.
Convolutional Layer Stack: Low parameter count, local views compose into a global view via depth.

What happens when we constrain width instead of connectivity? Hwang (2025) tackled deep, narrow Multi-Layer Perceptrons (MLPs), using a diffeomorphism approach to determine the absolute minimum width required for universal approximation. This work narrowed the gap between theoretical lower and upper bounds, proving exactly how narrow a network can be before it loses its ability to represent arbitrary coordinate transformations.

Finally, Kratsios (2021) unified many of these insights into a generalized topological framework. The author showed that if a network's activation function satisfies a property called transitivity (a property held by Leaky ReLU but not standard ReLU), the network retains its universal approximation power even under severe constraints, such as having highly sparse weight matrices or frozen final layers.

However, not all architectural constraints are fully understood. For instance, in Physics-Informed Neural Networks (PINNs), enforcing exact boundary conditions through the network's architecture—rather than via a penalty term in the loss function—is crucial for reliable convergence. Yet, as Müller et al. (2022) noted, whether these hard architectural boundary constraints fundamentally limit the network's capacity to approximate complex differential solutions remains an open theoretical question.

Summary

Architectural constraints are not inherently detrimental to a network's expressive capacity. Whether through the composition of local convolutions or the stacking of coordinate-wise affine coupling layers, deep networks successfully circumvent structural limitations. By understanding the underlying geometry—often by reframing the learning process as the composition of smooth, invertible diffeomorphisms—we can design highly efficient, constrained architectures that retain the full representational power of unconstrained models.

Key Foundational Work

Universality of Deep Convolutional Neural Networks (2018) — contributed 1 concept elements to this direction.
Approximation and non-parametric estimation of ResNet-type convolutional neural networks (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sparse_network_expressivity_7769cbe5,
  title = {sparse network expressivity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sparse-network-expressivity/rdr_7769cbe50b6b4406b3525ab23b04e675},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_sparse_network_expressivity_7769cbe5_v1,
  title = {sparse network expressivity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sparse-network-expressivity/rdr_7769cbe50b6b4406b3525ab23b04e675/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake