In the study of neural networks, the Universal Approximation Theorem (UAT) serves as a foundational guarantee: given enough neurons, a network can approximate any continuous function to arbitrary precision. For standard real-valued networks, this guarantee is remarkably easy to satisfy—the activation function simply needs to be non-polynomial. However, as we move into the complex domain to process data like radar signals or MRI scans, this simple condition no longer suffices.
Recent research has identified a more sophisticated criterion for complex-valued neural networks (CVNNs): the activation function must be non-polyharmonic.
The Failure of Polynomials and Their Complex Cousins
To understand why the non-polyharmonic condition is necessary, we must first look at why polynomials fail in the real-valued case. If an activation function $\sigma$ is a polynomial of degree $d$, then any linear combination of its neurons—which take the form $\sum w_i \sigma(v_i x + b_i)$—will also be a polynomial of degree $d$. Since polynomials of a fixed degree cannot approximate all continuous functions, the network's expressive power is strictly limited.
In the complex plane, the situation is more nuanced. As Voigtlaender demonstrates in The universal approximation theorem for complex-valued neural networks, the complex analog to a polynomial is a polyharmonic function. A function is polyharmonic if it satisfies the equation $\Delta^k f = 0$ for some integer $k$, where $\Delta$ is the Laplacian operator.
If a shallow CVNN uses a polyharmonic activation function, the entire network remains trapped within a specific, limited class of functions. Voigtlaender proves that being non-polyharmonic is the "if and only if" condition for universal approximation in shallow complex networks.
The Deep Network Distinction
When moving from shallow to deep architectures, the requirements for complex activation functions shift. Deep networks can compose functions, which allows them to break out of certain mathematical constraints that bind shallow networks.
However, Voigtlaender shows that deep CVNNs still face unique hurdles. For a deep network to be a universal approximator, the activation function must not only be non-polyharmonic but also avoid being purely holomorphic (complex-differentiable) or antiholomorphic. Holomorphic functions are extremely rigid; they are governed by the Cauchy-Riemann equations, which link their real and imaginary parts. If an activation function is holomorphic, the network essentially loses the ability to treat the real and imaginary components of the input with the necessary independence to approximate arbitrary complex functions.
From Possibility to Efficiency
Knowing that a network can approximate a function is different from knowing how efficiently it can do so. In Optimal approximation using complex-valued neural networks, Geuchen et al. establish that the non-polyharmonic property is also the key to achieving optimal approximation rates.
This means that for a wide class of smooth functions, CVNNs with non-polyharmonic activations can achieve the best possible trade-off between the number of parameters and the approximation error. This provides a theoretical "safety net" for practitioners, confirming that popular complex-valued architectures are not just heuristically useful but mathematically optimal.
A Unified View of Mathematical Richness
The search for these criteria is essentially a search for "mathematical richness." In the context of Residual Neural Networks (ResNets), Tabuada et al. propose a different but related perspective in Universal approximation power of deep residual neural networks via nonlinear control theory. By treating the layers of a ResNet as steps in a dynamical system, the authors show that universal approximation is possible if the activation function satisfies a specific quadratic differential equation.
This differential condition ensures that the network has enough "maneuverability" in its state space to reach any target output. Whether framed as non-polyharmonicity in the complex plane or as differential richness in ResNets, the core insight remains the same: universal approximation requires an activation function that can break the linear and structural symmetries of the space it inhabits.
Key Foundational Work
The universal approximation theorem for complex-valued neural networks (2022) — contributed 13 concept elements to this direction.
Universal approximation power of deep residual neural networks via nonlinear control theory (2021) — contributed 1 concept elements to this direction.