Theoretical Exploration of Flexible Transmitter Model

Jin-Hui Wu, Shao-Qun Zhang, Yuan Jiang, Zhi-Hua Zhou

Abstract

Neural network models generally involve two important components: network architecture and the neuron model. Although there are abundant studies about network architectures, only a few neuron models have been developed, such as the MP neuron model developed in 1943 and the spiking neuron model developed in the 1950s. Recently, a new bio-plausible neuron model, the Flexible Transmitter (FT) model, has been proposed. It exhibits promising behaviors, particularly on temporal-spatial signals, even when simply embedded into common feedforward network architectures. This paper attempts to understand the properties of the FT network (FTNet) theoretically. Under mild assumptions, we show that: i) FTNet is a universal approximator; ii) the approximation complexity of FTNet can be exponentially smaller than those of commonly-used real-valued neural networks with feedforward/recurrent architectures and is of the same order in the worst case; iii) any local minimum of FTNet is the global minimum, implying that it is possible to identify global minima by local search algorithms.

Introduction

Since the inception of the McCulloch-Pitts (MP) neuron in 1943, the field of artificial intelligence has largely focused on scaling network architectures—moving from simple perceptrons to complex Transformers and ResNets. However, the fundamental unit of these architectures, the neuron itself, has remained relatively stagnant. The MP neuron, which computes a weighted sum of inputs followed by a non-linear activation, is a significant oversimplification of biological processes.

To bridge this gap, the Flexible Transmitter (FT) model was recently introduced. Inspired by biological neurotrophic potentiation and depression, the FT model utilizes a two-variable formulation to mimic neurotransmitter dynamics. While early empirical evidence suggested that FT-based networks (FTNets) excel at processing spatio-temporal data, the mathematical justification for this performance remained unexplored. This paper provides the first rigorous theoretical foundation for FTNets, focusing on their approximation capabilities, computational complexity, and optimization landscape.

The Flexible Transmitter Framework

The FT model distinguishes itself by using complex-valued representations to handle the magnitude and phase of variables. This approach is particularly effective for data with antisymmetric or norm-preserving structures. The authors analyze two primary configurations:

  1. Feedforward FTNet (F-FTNet): A structure where FT neurons are arranged in layers without feedback loops.
  2. Recurrent FTNet (R-FTNet): A structure incorporating temporal dependencies.

A key technical innovation in this study is the use of the zReLU activation function, a complex-valued extension of the standard Rectified Linear Unit (ReLU), which facilitates the proof of complexity and approximation theorems.

Universal Approximation and Complexity

One of the primary questions for any new neural model is whether it can represent any given function. Through Theorems 1 and 2, the authors prove that a one-hidden-layer FTNet is a universal approximator. It can approximate any continuous function or discrete-time open dynamical system (DODS) on a compact set to arbitrary precision.

Beyond mere possibility, the paper addresses efficiency. The authors demonstrate a phenomenon known as exponential separation. For specific target functions, a standard Feedforward Neural Network (FNN) requires an exponential number of parameters ($\Omega(e^I)$) to achieve a certain accuracy. In contrast, an FTNet can achieve the same accuracy with only a polynomial number of parameters ($O(I^{15/4})$). This suggests that the FT model's intrinsic "local recurrent" properties allow it to capture temporal representations far more efficiently than standard architectures.

Importantly, the authors establish that FTNets are "safe" models: in the worst-case scenario, their complexity is of the same order as standard FNNs or RNNs, ensuring they are never significantly less efficient than traditional models.

The Optimization Landscape

Deep learning models are notoriously difficult to optimize due to non-convex loss surfaces filled with suboptimal local minima (traps). However, by leveraging Holomorphic Function Theory, the authors reveal a remarkably well-behaved optimization landscape for FTNets.

Under the assumption that training samples are linearly independent (a condition often met in high-dimensional regimes), the authors prove that for a wide class of analytic loss functions and holomorphic activations, every local minimum of an FTNet is a global minimum. This theoretical guarantee implies that local search algorithms, such as Gradient Descent, are mathematically guaranteed to find the global optimum, bypassing the common pitfalls of non-convex optimization.

Discussion and Limitations

While these theoretical results are groundbreaking, they come with specific caveats:

  • Sample Constraints: The proof for global optimality requires that the number of training samples does not exceed the input dimension. While this is common in certain high-dimensional tasks, it may not apply to all "big data" scenarios.
  • Activation Functions: The optimization results rely on complex-differentiable (holomorphic) properties. Standard real-valued activations like the original ReLU do not satisfy these requirements, necessitating the use of variants like zReLU.
  • Depth vs. Width: This research focuses primarily on one-hidden-layer "wide" networks. The theoretical advantages of multi-layer "Deep FTNets" and their generalization capabilities remain an open area for future investigation.

Conclusion

This work elevates the Flexible Transmitter model from an empirical curiosity to a mathematically robust alternative to the MP neuron. By proving universal approximation, exponential efficiency gains, and a trap-free optimization landscape, the authors provide a compelling case for the continued development of bio-plausible neuron models in modern AI.

Summary updated

Abstract

Neural network models generally involve two important components, i.e., network architecture and neuron model. Although there are abundant studies about network architectures, only a few neuron models have been developed, such as the MP neuron model developed in 1943 and the spiking neuron model developed in the 1950s. Recently, a new bio-plausible neuron model, flexible transmitter (FT) model (Zhang and Zhou, 2021), has been proposed. It exhibits promising behaviors, particularly on temporal–spatial signals, even when simply embedded into the common feedforward network architecture. This article attempts to understand the properties of the FT network (FTNet) theoretically. Under mild assumptions, we show that: 1) FTNet is a universal approximator; 2) the approximation complexity of FTNet can be exponentially smaller than those of commonly used real-valued neural networks with feedforward/recurrent architectures and is of the same order in the worst case; and 3) any local minimum of FTNet is the global minimum, implying that it is possible to identify global minima by local search algorithms.
BibTeX
@article{wu2021theoretical,
  author = {Jin-Hui Wu and Shao-Qun Zhang and Yuan Jiang and Zhi-Hua Zhou},
  title = {{Theoretical Exploration of Flexible Transmitter Model}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2021},
  doi = {10.48550/arxiv.2111.06027},
}
Figure