Towards Theoretical Understanding of Flexible Transmitter Networks via Approximation and Local Minima

Jin-Hui Wu, Shao-Qun Zhang, Yuan Jiang, Zhi-Hua Zhou

Abstract

Neural network research has historically focused on architectural innovations, such as Transformers or Convolutional Neural Networks, while the underlying neuron model has remained largely stagnant since the McCulloch-Pitts (MP) model of 1943. The recently proposed Flexible Transmitter (FT) neuron model offers a bio-plausible alternative that leverages complex-valued mathematics to mimic neurotrophic potentiation and depression. While FT Networks (FTNets) have shown empirical promise in processing temporal-spatial signals, their theoretical properties have remained unexplored. This paper establishes the first mathematical foundation for FTNets, proving they are universal approximators with the potential for exponential efficiency over standard real-valued networks. Furthermore, we demonstrate that the loss landscape of feedforward FTNets is devoid of suboptimal local minima under mild conditions, facilitating global optimization.

Introduction

In the landscape of deep learning, the fundamental building block—the neuron—has seen remarkably little evolution compared to the architectures that house them. Most modern networks still rely on the MP neuron, which simplifies biological signaling into a weighted sum followed by a non-linear activation. The Flexible Transmitter (FT) model was introduced to bridge this gap, incorporating complex-valued dynamics to better represent biological signal transmission.

Despite empirical successes, the lack of theoretical rigor has hindered the broader adoption and understanding of FTNets. This research addresses this void by investigating three critical areas: expressivity (Universal Approximation), efficiency (Approximation Complexity), and optimizability (Loss Landscape).

Universal Approximation

The authors first establish that FTNets possess the necessary expressivity to serve as general-purpose models. By employing measure theory and complex analysis, the paper proves two primary theorems:

  1. Feedforward FTNets (F-FTNet): A one-hidden-layer F-FTNet can approximate any continuous function on a compact set to arbitrary precision.
  2. Recurrent FTNets (R-FTNet): An R-FTNet can approximate any discrete-time open dynamical system.

These results confirm that FTNets are not merely specialized tools for niche tasks but are mathematically equivalent in potential to standard Feedforward Neural Networks (FNNs) and Recurrent Neural Networks (RNNs).

Approximation Complexity: The Case for Efficiency

The most significant contribution of this work lies in the comparison of "approximation complexity"—the number of neurons required to achieve a specific level of accuracy. The authors identify a "complexity separation" between FTNets and traditional real-valued networks.

Exponential Advantage

The paper proves that there exist specific target functions where standard FNNs or RNNs require an exponentially large number of hidden neurons to achieve a low approximation error. Specifically, for an input dimension $I$, a standard network might require a width of $\Omega(e^{cI})$. In contrast, an FTNet can approximate these same functions using only a polynomial number of neurons, $O(I^{15/4})$. This suggests that for certain classes of data, FTNets are fundamentally more efficient at representation.

Worst-Case Parity

Critically, this efficiency does not come at the cost of a lower bound. The authors prove a "safety net" theorem: in the worst-case scenario, any function that can be approximated by a standard FNN or RNN can also be approximated by an FTNet using the same order of magnitude of hidden neurons ($O(H)$). Thus, FTNets offer a significant potential upside with no theoretical downside in terms of parameter scaling.

The Optimization Landscape

A perennial challenge in deep learning is the presence of suboptimal local minima, which can trap gradient-based optimization algorithms. The authors analyze the empirical loss surface of F-FTNets using the properties of holomorphic (complex differentiable) functions.

They prove that for an F-FTNet utilizing a holomorphic, non-polynomial activation function (such as zReLU), the loss landscape is remarkably well-behaved. Under the condition that training samples are linearly independent, the paper demonstrates that every local minimum is a global minimum. If the network has not reached a loss of zero, a mathematical path always exists to further decrease the loss, implying that local search algorithms like Stochastic Gradient Descent (SGD) are theoretically capable of finding the global optimum.

Limitations and Future Outlook

While this work provides a robust theoretical starting point, it focuses primarily on shallow (one-hidden-layer) architectures. The mathematical behavior of deep FTNets—specifically how they transform feature spaces across multiple layers—remains an open question. Additionally, the proof regarding the absence of suboptimal local minima currently applies only to the feedforward variant (F-FTNet), leaving the landscape of Recurrent FTNets for future investigation.

By providing these proofs of universal approximation, exponential efficiency, and global optimizability, this research elevates the FT neuron from a bio-inspired heuristic to a mathematically grounded alternative for modern machine learning.

Summary updated

Abstract

Neural network models generally involve two important components, i.e., network architecture and neuron model. Although there are abundant studies about network architectures, only a few neuron models have been developed, such as the MP neuron model developed in 1943 and the spiking neuron model developed in the 1950s. Recently, a new bio-plausible neuron model, Flexible Transmitter (FT) model, has been proposed. It exhibits promising behaviors, particularly on temporal-spatial signals, even when simply embedded into the common feedforward network architecture. This paper attempts to understand the properties of the FT network (FTNet) theoretically. Under mild assumptions, we show that: i) FTNet is a universal approximator; ii) the approximation complexity of FTNet can be exponentially smaller than those of commonly-used real-valued neural networks with feedforward/recurrent architectures and is of the same order in the worst case; iii) any local minimum of FTNet is the global minimum, implying that it is possible to identify global minima by local search algorithms.
BibTeX
@article{wu2021towards,
  author = {Jin-Hui Wu and Shao-Qun Zhang and Yuan Jiang and Zhi-Hua Zhou},
  title = {{Towards Theoretical Understanding of Flexible Transmitter Networks via Approximation and Local Minima}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2021},
  doi = {10.48550/arxiv.2111.06027},
}
Figure