Fractional Order Dynamics for Long Memory Neural Networks

This direction is split into other directions: non-local state integration. Its last version stays readable and citable here.

Version 1 of 118 papers35 conceptsfrozen 2026-10-04

Most neural networks are fundamentally "forgetful." Whether they are standard Feedforward Networks or Spiking Neural Networks (SNNs), their underlying mathematics usually relies on integer-order differential equations. In these systems, the future state depends only on the immediate present—a property known as Markovian dynamics. While this makes them computationally efficient, it forces them to model the past using exponential decay, where information vanishes rapidly over time.

A growing body of research suggests that replacing these standard equations with fractional-order dynamics allows neurons to escape this "short-term memory" trap. By using fractional calculus, researchers are building networks that naturally exhibit power-law memory, enabling them to integrate information across vast timescales without the need for complex gating mechanisms or massive parameter counts.

The Limits of Exponential Decay

In a traditional Leaky Integrate-and-Fire (LIF) neuron, the membrane potential decays exponentially. This is a "local" operation: to know the next state, you only need to know the current state and the current input. However, as A. Gal, Shimon Marom, and colleagues demonstrated in Dynamics of Excitability over Extended Timescales in Cultured Cortical Neurons, real biological neurons don't work this way. They exhibit "long-memory" dynamics where a cell's excitability is statistically influenced by activity that occurred hours earlier.

Standard models fail to capture these scale-invariant fluctuations because they treat slow changes as simple sums of fast processes. To truly mimic biological realism and capture complex temporal patterns, we need a mathematical framework where the "derivative" of a state isn't just a local slope, but a weighted integral of its entire history.

How Fractional Derivatives Create Memory

Fractional calculus generalizes the concept of a derivative to non-integer orders (e.g., a "0.5th" derivative). Unlike standard derivatives, a fractional derivative is a non-local operator.

As explained by Chengjie Ge and Yufeng Peng in their work on Fractional-order Spiking Neural Networks, this transition replaces exponential decay with power-law relaxation. In a power-law system, the "tail" of the memory is much heavier; distant past events maintain a persistent, albeit small, influence on the current state.

This mechanism allows a single neuron to represent a continuum of timescales. In Capturing multiple timescales of adaptation, Adrienne Fairhall and Kenneth W. Latimer show that by extending history filters to span several seconds, models can replicate sophisticated adaptive behaviors like fractional differentiation—where a system responds to the rate of change of input variance rather than just the input itself.

Applications Across Domains

The "long-memory" property of fractional dynamics has proven transformative across several distinct subfields of AI:

  • Computer Vision: In FOCNet, Xixi Jia and Lei Zhang model deep networks as fractional dynamic systems. This allows a layer to incorporate information from all preceding layers as a weighted sum, preserving high-frequency structural details during image denoising without the parameter explosion of dense concatenations.
  • Graph Robustness: Qiyu Kang and Kai Zhao introduced FROND, which uses fractional-order continuous dynamics to make Graph Neural Networks more robust. Because the evolution of a node's representation is constrained by its entire historical trajectory, the model is significantly harder to "shock" with adversarial attacks.
  • Neuroscience and Medicine: Paul Bogdan and Yaoyue Wang have used these models to suppress epileptic seizures. They found that during a seizure, brain activity becomes hyper-dependent on its own history. By modeling this with fractional dynamics, they can design controllers that stabilize the network and reduce signal amplitude.
  • Physical Systems: For systems with inherent "lag" or diffusion, such as batteries, Lujuan Dang demonstrated that Physics-Informed Neural Networks (PINNs) using fractional derivatives can more accurately estimate a battery's state of charge by capturing the long-range memory of electrochemical processes.

Overcoming the Computational Burden

The primary challenge of fractional-order models is that they are computationally expensive; calculating a derivative that depends on the "entire history" usually requires storing every previous state.

Recent breakthroughs have made these models practical for large-scale deep learning. Qiyu Kang and Xuhao Li developed an efficient adjoint backpropagation method for training these networks. By solving an augmented fractional differential equation backward in time, they can compute gradients without storing the full forward trajectory, reducing memory overhead by up to 60%.

Similarly, in the realm of spiking neurons, Chengjie Ge and colleagues implemented f-SNNs using predictor-corrector numerical methods. This allows the network to approximate fractional derivatives in discrete time, giving spiking neurons a "heavy-tailed" memory that significantly improves their performance on complex time-series tasks.

By moving from integer to fractional orders, researchers are essentially giving neural networks a sense of time that mirrors the complexity of the physical and biological world. These models don't just process sequences; they inhabit a dynamical space where the past is never truly gone, but continues to shape the present through a mathematically elegant memory kernel.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_fractional_order_dynamics_for_long_memor_8f223343,
  title = {Fractional Order Dynamics for Long Memory Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fractional-order-dynamics-for-long-memory-neural-networks/rdr_8f223343be0d432a9078d6fc8a5147e7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_fractional_order_dynamics_for_long_memor_8f223343_v1,
  title = {Fractional Order Dynamics for Long Memory Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fractional-order-dynamics-for-long-memory-neural-networks/rdr_8f223343be0d432a9078d6fc8a5147e7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}