In the evolving landscape of machine learning, Functional Data Analysis (FDA) has emerged as a critical field where inputs are not discrete vectors, but continuous functions—such as time-series signals, spectral data, or spatial weather patterns. While deep learning has seen empirical success in these domains, the theoretical underpinnings of how Rectified Linear Unit (ReLU) networks approximate these infinite-dimensional mappings have remained elusive. This paper provides a rigorous mathematical foundation for the approximation of nonlinear functionals using deep ReLU networks, establishing convergence rates that match theoretical lower bounds.
The Challenge of Functional Approximation
A functional is a mapping from a function space (like $L^p$) to the real numbers. While the Universal Approximation Theorem guarantees that neural networks can approximate continuous functionals, most existing proofs rely on smooth activation functions (e.g., sigmoids or tanh). ReLU, however, is piecewise linear and non-differentiable at zero, which complicates standard analytical tools. This research bridges that gap by demonstrating how the geometric properties of ReLU networks can be leveraged to approximate complex functionals efficiently.
Constructive Proof Strategy
The authors employ a three-stage constructive proof to demonstrate the approximation power of ReLU networks:
1. Discretization via Legendre Polynomials
To handle infinite-dimensional inputs, the functional space is first mapped into a finite-dimensional polynomial space. The authors use a continuous linear operator based on Legendre polynomials to transform the input function $f$ into a finite vector of coefficients. This step effectively reduces the problem to a high-dimensional vector approximation task while controlling the information loss.
2. High-Dimensional Triangulation
The resulting finite-dimensional space is partitioned into "simplexes" (the multi-dimensional generalization of triangles). By creating a triangulation of the domain, the authors can define piecewise linear interpolations over the space.
3. ReLU Implementation and the "Min" Trick
A pivotal contribution of this work is the explicit construction of "spike functions" using ReLU layers. A spike function $\psi$ is defined to be 1 at a specific vertex of a simplex and 0 at all others. The authors prove that these functions—and the resulting piecewise linear interpolations—can be exactly represented by a deep ReLU network.
Crucially, they demonstrate that the min(x_1, ..., x_d) operation, which is essential for defining the geometry of simplexes, can be efficiently implemented using a ReLU network with $d-1$ hidden layers. This allows the network to "logic" its way through the geometric partitions of the input space.
Key Theoretical Results
The paper establishes two primary theorems regarding the error bounds of these approximations. The total error is decomposed into the discretization error (from the polynomial mapping) and the interpolation error (from the network's approximation of the mapping).
For functionals belonging to a Hölder space with smoothness parameter $\beta$, the authors establish an approximation rate of:
$$|F(f) - \Theta_v(f)| = O\left( \left( \frac{\log M}{\log(\log M)} \right)^{-\frac{\beta\lambda}{s}} \right)$$
where $M$ represents the number of nonzero weights in the network. This result is significant because it matches the lower bounds established in previous literature for much smoother activation functions, proving that ReLU networks are theoretically optimal for functional approximation tasks.
Implications and Limitations
This work confirms that despite its non-differentiability, the ReLU activation is as powerful as smooth activations for functional data, provided the network architecture is sufficiently deep to handle the required geometric partitions.
However, the authors acknowledge the "curse of dimensionality." As the dimensionality of the discretized space increases, the number of simplexes required for triangulation grows exponentially. Furthermore, this is an existence proof; while it proves such a network exists and defines its structure, it does not address the dynamics of training such a network via stochastic gradient descent (SGD) in practice.
Conclusion
By providing a rigorous link between ReLU networks and functional approximation theory, this paper offers a theoretical justification for the use of deep learning in FDA. Future work may extend these results to more specialized architectures, such as Convolutional Neural Networks (CNNs), and explore the generalization capabilities of these networks on real-world functional datasets.
Summary updated