When we model deep neural networks as continuous dynamical systems—such as Neural Ordinary Differential Equations (Neural ODEs) or continuous Graph Neural Networks (GNNs)—we typically rely on standard integer-order calculus. In these frameworks, the state of a system evolves according to a local derivative: the rate of change $\frac{dx}{dt}$ depends strictly on the current state $x(t)$.
This formulation introduces a fundamental limitation: it is inherently Markovian. Because the derivative only looks at an infinitesimal window around the present moment, any historical information must be compressed entirely into the current state vector. In linear systems, this causes historical influence to decay exponentially fast. To capture long-range temporal dependencies or complex hereditary traits, standard architectures must resort to high-dimensional hidden states or intricate, parameter-heavy gating mechanisms.
A growing body of research solves this problem by replacing integer-order derivatives with fractional-order derivatives. By expanding calculus to non-integer orders (such as a derivative of order $\alpha = 0.5$), neural networks gain an inherent, mathematically elegant long-term memory.
How Fractional Derivatives Embed History
To understand why fractional-order dynamics naturally possess memory, we must look at how a fractional derivative is defined. Unlike standard derivatives, which are purely local operators, fractional derivatives are non-local.
Consider the Caputo fractional derivative, a common choice in neural modeling highlighted by Zeng-bao Wu et al. in A New System of Global Fractional-order Interval Implicit Projection Neural Networks. The fractional derivative of a state $x(t)$ is defined as an integral over the system's entire historical trajectory, weighted by a power-law kernel:
$$D^\alpha x(t) = \frac{1}{\Gamma(1-\alpha)} \int_0^t (t-\tau)^{-\alpha} \frac{dx(\tau)}{d\tau} d\tau$$
Because of the term $(t-\tau)^{-\alpha}$, the current rate of change is explicitly a function of every state the system has visited since $t=0$.
This mathematical structure provides two crucial intuitive properties:
- Non-local Integration: The system cannot update its current state without accounting for its historical path. The evolution of the network's representations is constrained by its past.
- Power-Law Decay: Unlike the exponential forgetting typical of standard differential equations, the power-law kernel decays slowly. This heavy-tailed memory allows distant past events to maintain a persistent, non-negligible influence on the present state.
As Emily A. Reed et al. note in Fractional cyber-neural systems - a brief survey, biological systems like the human brain naturally exhibit power-law relaxation and fractal dynamics. Standard integer-order models assume a memoryless nature that fails to capture these biological realities, whereas fractional-order systems match them natively.
Transforming Neural Architectures
By embedding fractional calculus into neural networks, researchers have unlocked new capabilities across a wide variety of architectures.
Recurrent and Associative Networks
In traditional recurrent networks, memory is maintained by recurrent weight matrices. However, in a fractional-order recurrent network, memory is built directly into the state space itself. As demonstrated by Yi‐Fei Pu et al. in Fractional Hopfield Neural Networks: Fractional Dynamic Associative Recurrent Neural Networks, the equilibrium points and attractors of a network are determined not just by the connection weights, but also by the specific fractional order assigned to individual neurons. This provides a highly tunable state space for associative memory tasks.
Continuous Graph Neural Networks
Continuous GNNs model node feature propagation as a continuous diffusion process. A notorious issue here is oversmoothing, where features across different nodes collapse and become indistinguishable as the network deepens.
To combat this, Qiyu Kang et al. introduced the FROND (FRactional-Order graph Neural Dynamical network) framework in Unleashing the Potential of Fractional Calculus in Graph Neural Networks with FROND. By forcing feature updates to depend on the entire history of node states, FROND prevents rapid convergence to a uniform state. Furthermore, a companion study by Qiyu Kang et al., Coupling Graph Neural Networks with Fractional Order Continuous Dynamics: A Robustness Study, proved that this historical constraint makes the network significantly more robust against adversarial attacks on both graph structure and node features.
Spiking Neural Networks
Standard Spiking Neural Networks (SNNs) rely on leaky integrate-and-fire models where the membrane potential decays exponentially. This makes them inherently "forgetful." In Fractional-Order Spiking Neural Network, Chengjie Ge et al. replaced these first-order equations with fractional derivatives. The resulting $f$-SNNs exhibit power-law relaxation, allowing a single neuron to maintain a rich continuum of multiple timescales and capture long-range temporal correlations without extra parameters.
Deep Feedforward Networks
The benefits of fractional dynamics extend even to feedforward tasks like image denoising. In FOCNet: A Fractional Optimal Control Network for Image Denoising, Xixi Jia et al. modeled deep network propagation as a fractional-order dynamic system. Instead of a standard residual network where layer $l+1$ only receives input from layer $l$, the fractional framework allows each layer to naturally integrate information from all preceding layers. This power-law memory effect preserves high-frequency structural details and complex textures that are typically lost in deep architectures.
Standard Residual Block (Integer-Order):
[Layer 1] ----> [Layer 2] ----> [Layer 3] ----> [Layer 4]
(Only immediate predecessor information is passed)
Fractional-Order Propagation (Non-Local):
[Layer 1] ----+--------------+--------------+----> [Layer 4]
| | |
v | |
[Layer 2] -----------+ |
v |
[Layer 3] -----------+
(Current layer integrates a weighted history of ALL preceding layers)
Real-World Systems and Computational Efficiency
The primary real-world application of this framework lies in understanding biological neural systems. In works like Minimum number of probes for brain dynamics observability by Sérgio Daniel Pequito et al. and Re-thinking EEG-based non-invasive brain interfaces: modeling and analysis by Gaurav Gupta et al., treating brain activity as a fractional-order dynamical system drastically improves state prediction and motor task classification from EEG signals. It allows models to account for the brain's natural long-range temporal dependencies and spatial interactions. Even when dealing with unmeasured external stimuli—the "unknown unknowns"—Gaurav Gupta et al. showed in Dealing with Unknown Unknowns: Identification and Selection of Minimal Sensing for Fractional Dynamics with Unknown Inputs that fractional models allow for the full reconstruction of hidden states using a minimal number of physical sensors.
Overcoming the Computational Bottleneck
If fractional-order dynamics are so powerful, why haven't they completely replaced standard models? The bottleneck is computational. Because computing a fractional derivative requires integrating over the entire history of the system, standard backpropagation requires storing the full forward trajectory in memory. For a trajectory of length $T$, this leads to a prohibitive $O(T^2)$ memory overhead.
This scaling bottleneck was directly addressed by Qiyu Kang et al. in Efficient Training of Neural Fractional-Order Differential Equation via Adjoint Backpropagation. The authors developed an adjoint sensitivity method tailored for fractional differential equations. By deriving and solving an augmented fractional differential equation backward in time, the model can compute gradients without storing the intermediate states of the forward pass. This technique reduces memory requirements by up to 60%, making the training of large-scale, history-aware fractional neural models mathematically and practically viable.
Key Foundational Work
Fractional Hopfield Neural Networks: Fractional Dynamic Associative Recurrent Neural Networks (2016) — contributed 3 concept elements to this direction.
A New System of Global Fractional-order Interval Implicit Projection Neural Networks (2016) — contributed 2 concept elements to this direction.
Minimum number of probes for brain dynamics observability (2015) — contributed 1 concept elements to this direction.