Abstract
To better understand the approximation capabilities of various neural network architectures, this paper investigates the universal approximation property (UAP) across a broad scope of function spaces. We characterize universal approximators as architectures that implicitly decompose function spaces into topologically regular subspaces where a transitive dynamical system describes the model's structure. This framework provides a simple criterion for constructing universal approximators as transformations of standard feed-forward architectures. Notably, we demonstrate that activation functions such as Leaky-ReLU, which satisfy topological transitivity, enable expressivity through depth by mixing function neighborhoods—a property standard ReLU lacks. Finally, we introduce a modified architecture capable of approximating continuous functions uniformly across entire domain spaces, overcoming the limitations of traditional compact-set approximation.
Introduction
The Universal Approximation Theorem is a cornerstone of deep learning theory, asserting that neural networks can approximate any continuous function to arbitrary precision. However, since the late 1980s, these proofs have largely been conducted on a case-by-case basis for specific architectures. This fragmented approach complicates the design of novel, theoretically grounded models and limits our understanding of how constraints—such as weight sparsity or fixed output layers—impact a network's fundamental capabilities.
The Limitations of Current Theory
Standard UAP results typically apply only to functions defined on "compact sets" (bounded, closed regions). In practical applications, however, we often require models that behave predictably as inputs trend toward infinity. Moreover, existing theory often fails to explain why certain activation functions perform better in deep architectures or how sparsity affects the fundamental limit of what a network can learn. This paper seeks a general theory of UAP that works across many different function spaces and provides a recipe for building models that approximate functions over the entire input space.
A Unified Theory via Topological Transitivity
This research shifts the paradigm by treating neural architectures through the lens of functional analysis and dynamical systems. The author proves that an architecture is a universal approximator if and only if it can "mix" the function space, a property known as topological transitivity.
In this context, the function space is treated as a Fréchet space. Topological transitivity describes how a system's state eventually visits every neighborhood of the space—much like how stirring a pot of paint eventually distributes a single drop of color throughout the entire container. Using this pillar, the paper establishes that UAP is mathematically equivalent to the architecture containing the orbit of a topologically transitive dynamical system.
Activation Functions and Expressivity through Depth
A significant insight of this work is the role of the activation function $\sigma$. For a deep network to maintain expressivity, $\sigma$ should ideally be injective (one-to-one) and have no fixed points (meaning $\sigma(x) \neq x$).
- Leaky ReLU vs. ReLU: The standard ReLU function fails this criterion because it has an infinite number of fixed points for all $x > 0$ (where $ReLU(x) = x$). This "stagnation" prevents deeper layers from effectively mixing the input space.
- Transitivity: In contrast, a slightly shifted and rescaled Leaky ReLU (e.g., $1.1x + 0.1$ for $x \geq 0$) satisfies the transitivity requirement.
For these transitive activation functions, the paper obtains improved approximation rates described in terms of network breadth and depth. This allows networks to maintain their approximation power even when the initial layers are sparse or when the final layers are forced to follow specific pre-trained constraints.
Global Approximation Capabilities
Beyond local approximation, the paper introduces a modified feed-forward architecture that achieves global approximation. While most neural network theory only guarantees accuracy on a bounded subset of $\mathbb{R}^n$, this construction can approximate continuous functions with "non-pathological growth" uniformly across the entire Euclidean space. The results show that while standard feed-forward architectures typically cannot achieve this global reach, simple modifications to the transformation process can enable it.
Conclusion
By providing a unified mathematical "toolbox," this work moves beyond ad-hoc proofs. It establishes a rigorous link between the topological properties of activation functions and the global capabilities of deep networks. While identifying the optimal transformation for specific practical tasks remains an open challenge, this framework provides the theoretical foundation necessary to design the next generation of expressive, constrained, and globally accurate neural architectures.
Summary updated