Universal Approximation on Unbounded Domains

27 concepts18 papers2 proposals

The foundational guarantee of deep learning is the Universal Approximation Theorem: given enough neurons, a neural network can approximate any continuous function to arbitrary precision. However, if you look closely at the mathematical fine print of classical proofs, you will almost always find a crucial caveat: "Let $K$ be a compact subset of $\mathbb{R}^d$."

In mathematics, compactness means the domain is closed and bounded. In practice, this means classical theory only guarantees a network will behave well within a predefined, finite box. If your inputs scale toward infinity—such as financial market returns, unbounded sensor noise, or principal component scores of continuous functional data—traditional guarantees completely break down.

As a result, a growing body of research focuses on unbounded domain approximation: extending the mathematical and structural guarantees of neural networks to the entire infinite real space.

The Compactness Trap

Why does the assumption of a bounded domain matter so much? To build intuition, imagine trying to approximate the simple polynomial $f(x) = x^2$ across the entire real line using a standard network.

If we restrict our view to a compact interval like $[-1, 1]$, a combination of a few rectified linear units (ReLUs) or sigmoids can easily mimic the curve. But as $x$ grows toward infinity, a standard network made of sigmoids will flatten out, completely failing to track the target function's polynomial growth. Even standard ReLU networks, which grow linearly, cannot inherently match higher-order polynomial growth at infinity without an infinite number of neurons.

This theoretical gap creates real-world vulnerabilities. As noted by Unser, traditional theoretical frameworks that assume compact sets limit our mathematical understanding of models operating on unbounded domains. This limitation is felt across various architectures:

  • Generative Models: Theoretical proofs for Mixture of Experts (MoE) models (Nguyen et al.) and Deep Belief Networks (Fischer et al.) frequently rely on compact support assumptions, leaving a gap when modeling real-world variables that span infinite ranges.
  • Functional Data Analysis: When deep networks process functional data (data represented as continuous curves), the inputs are often decomposed into principal component scores. As Zhou et al. point out, these scores are inherently unbounded, violating core assumptions of standard deep learning theory.
  • Geometric Architectures: Even specialized frameworks like Deep Sets (Zaheer et al.) face theoretical hurdles when extending universal approximation guarantees from fixed-size or countable domains to variable-sized sets in uncountable, infinite spaces.

Mechanism 1: Aligning Activations and Weighted Norms

To prove that a network can approximate a function globally across an infinite space, mathematicians cannot use standard uniform convergence norms, which blow up at infinity. Instead, they must align the growth rate of the network's activation functions with the growth rate of the target function space.

In a seminal step, Hatano et al. (2020) proved that two-layer ReLU networks can achieve global universality over the entire real line. The key insight was to introduce a specialized function space and norm that explicitly accounts for linear growth at infinity. Because the ReLU activation function itself grows linearly, the network's inherent architectural constraints perfectly match the target function's boundary behavior.

Target Function Space (Growth at Infinity) <---> Specialized Norm <---> Network Activation Growth

When functions grow faster than a straight line, standard ReLU is no longer enough. To handle this, Hatano et al. (2024) extended this framework to $k$-Rectified Linear Units ($k$-ReLU) for $k > 1$ (e.g., $f(x) = \max(0, x)^k$). By carefully constructing norms that control polynomial growth rates at infinity, the authors proved that $k$-ReLU networks achieve global universality in spaces designed for unbounded functions.

Alternatively, if the target distribution naturally decays at infinity rather than growing, approximation is achievable via measure theory. For instance, Baptista et al. demonstrated that four-layer ReLU architectures can successfully extend universal approximation to infinite Euclidean spaces under Gaussian measures, leveraging the fact that the underlying probability space decays predictably at the boundaries.

Mechanism 2: Architectural Modifications

Beyond changing the mathematical norms used for proofs, researchers have designed explicit architectural mechanisms to handle infinite domains.

Growth-Controlling Wrappers

Instead of modifying the internal layers of a network, one can modify how the network interacts with the boundary of the infinite space. Kratsios showed that standard neural architectures can be extended to approximate functions across the entire infinite Euclidean space by applying "growth-controlling wrappers." These wrappers act as mathematical guards around the network, managing its behavior at the outer fringes of the input space to ensure uniform approximation properties globally.

Recursive Weight Modulation

To capture complex polynomial functions globally without exploding parameter counts, Gabbay introduced Fibred Neural Networks (fNNs). In this recursive architecture, "parent" neurons dynamically modulate the weights of "child" sub-networks. This dynamic weight modulation allows the network to represent high-order polynomial relationships that remain valid even as inputs scale toward infinity, bypassing the local limitations of standard feedforward layers.

Gridless Splines

The limitation of bounded domains is particularly acute in Kolmogorov-Arnold Networks (KANs), which replace fixed weights with learnable univariate spline functions. As Pozdnyakov et al. observe, standard spline-based activations are constrained to predefined, bounded grids; they fail completely when encountering outliers or unbounded activations common in deep layers.

To solve this, Moradzadeh et al. developed Unbounded Kolmogorov-Arnold Networks (UKANs). By removing fixed, bounded grids entirely, UKANs allow spline-based architectures to approximate functions over unbounded domains, unlocking scalability for large-scale deep learning tasks.

Standard KAN:  [Input] ---> [Fixed Bounded Grid Spline] ---> (Fails on Outliers)
Unbounded KAN: [Input] ---> [Gridless/Adaptive Spline]  ---> (Succeeds Globally)

The Theoretical Frontier

While significant progress has been made, several open challenges remain at the boundary of infinite-domain expressivity.

One major friction point is network width. Recent breakthroughs have established the absolute minimum width required for a network to remain a universal approximator. For example, Cai (2023) proved that a network using a "Universal Ordering of Extrema" activation function requires a minimum width of exactly $\max(d_x, d_y)$, while Shin et al. generalized tight width bounds ($\max\{d_x, d_y, 2\}$) to a broad class of "squashable" activations like Sigmoid and Swish. However, both Cai and Shin et al. note that these tight width bounds are strictly proven for compact domains. Whether these minimal widths are sufficient to maintain universal approximation across the entire unbounded real coordinate space remains an open question.

Furthermore, moving to unbounded domains alters the fundamental relationship between neural networks and other machine learning paradigms. On compact domains, neural networks and kernel methods are often seen as closely related or interchangeable. However, Kumar 0010 et al. exposed a surprising theoretical gap: on the unbounded domain of $\mathbb{R}^d$, there are functions that a Gaussian Reproducing Kernel Hilbert Space (RKHS) can represent easily, but which a shallow neural network cannot represent at all because the network's required norm becomes infinite.

Similarly, finite mapping approaches—such as the linguistic "vocabulary" composition framework proposed by Cai (2024)—remain mathematically locked to compact domains. Bridging these gaps is the next major step in establishing a truly global theory of deep learning.

Key Foundational Work

Fibring Neural Networks (2004) — contributed 2 concept elements to this direction.
Fewer Epistemological Challenges for Connectionism (2005) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_unbounded_domain_approximation_e37492b8,
  title = {unbounded domain approximation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unbounded-domain-approximation/rdr_e37492b8ecdf44e4a4c58321375fc600},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_unbounded_domain_approximation_e37492b8_v1,
  title = {unbounded domain approximation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unbounded-domain-approximation/rdr_e37492b8ecdf44e4a4c58321375fc600/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake