Deep neural networks, sequence models, and complex dynamical systems are highly effective at processing temporal data, but their inner workings are often treated as black boxes. When a model successfully predicts the next frame of a video, the trajectory of a physical system, or a phrase in a speech signal, it is difficult to isolate exactly how it combines past information to form its output.
To unpack these complex systems, researchers are increasingly turning to a classical mathematical tool: the Volterra series expansion.
The core intuition behind a Volterra series is that it acts as a functional extension of a Taylor series for systems with memory. While a standard Taylor series decomposes a static nonlinear function into a polynomial sum ($x, x^2, x^3, \dots$), a Volterra series decomposes a nonlinear dynamical system into a hierarchical sum of multidimensional convolutions. By mapping modern architectures to this framework, we can translate opaque, deeply stacked nonlinear layers into an explicit, interpretable hierarchy of input correlations.
The Intuition: Decomposing Memory into Higher-Order Kernels
In a purely linear dynamical system, the relationship between an input signal $u(t)$ and an output $y(t)$ is fully described by a standard convolution with a single impulse response, or kernel. However, real-world systems involve nonlinear interactions over time. For example, the output might depend on what happened one second ago multiplied by what happened two seconds ago.
A Volterra series handles this by breaking the system's response down into a series of structured terms:
- First-order kernel: A standard linear convolution capturing how individual past inputs independently affect the present.
- Second-order kernel: A two-dimensional convolution capturing pairwise interactions between past inputs at different time lags.
- Third-order kernel: A three-dimensional convolution capturing triplet interactions, and so on.
Input Signal u(t) ───► [ 1st-Order Kernel (Linear) ] ───┬─► Sum ───► Output y(t)
───► [ 2nd-Order Kernel (Pairs) ] ───┼─┤
───► [ 3rd-Order Kernel (Triplets)] ───┘
This mathematical decomposition allows us to analyze complex processing stages systematically. For instance, in biological vision, Juyue Chen et al. demonstrated in Asymmetric ON-OFF processing of visual motion cancels variability induced by the structure of natural scenes that fruit flies rely on third-order motion processing kernels to cancel out scene-induced noise that second-order (pairwise) estimates cannot resolve. By isolating these specific kernel orders, the authors could precisely pinpoint how the fly's visual system handles natural scene statistics.
Opening the Black Box of Deep Convolutional Networks
Why does this classical framework matter for modern deep learning? Because deep convolutional architectures can be mathematically rewritten as Volterra series. Stacking convolutional layers interspersed with nonlinear activation functions (like ReLU or Sigmoid) implicitly constructs higher-order input interactions.
By taking Taylor expansions of these activation functions, researchers can explicitly map deep networks to classical system identification structures. Carl R. Andersson et al. proved in Deep Convolutional Networks in System Identification that temporal convolutional networks (TCNs) are mathematically equivalent to infinite-degree Volterra series and parallel Wiener models. This equivalence provides a solid theoretical bridge, showing that deep learning sequence models are not entirely new paradigms, but rather highly efficient evolutions of traditional nonlinear operator approximations.
Building on this, Tenghui Li et al. showed in Toward Understanding Convolutional Neural Networks from Volterra Convolution Perspective that common CNN structures—such as Conv-Act-Conv blocks—can be well-approximated by finite-term Volterra convolutions. This transformation replaces the black-box layers of a neural network with interpretable "proxy kernels," making the network's behavior and robustness amenable to established nonlinear system theory.
Structuring Memory and Sequence Models
For recurrent architectures and State Space Models (SSMs), tracking long-term dependencies safely without exploding gradients is a persistent challenge. Volterra series expansions offer a way to analyze and explicitly structure how these models retain information.
In the context of modern state space architectures, Jack Goffinet et al. demonstrated in HiPPO Zoo: Explicit Memory Mechanisms for Interpretable State Space Models that nonlinear interactions within sequence models can be decomposed into interpretable Volterra kernels. By replacing black-box nonlinear readouts with this structured representation, it becomes possible to visualize and analyze exactly how higher-order dependencies are maintained within the state space.
This perspective unifies modern sequence models with older, foundational theories of time-delay networks. For example, Bert de Vries et al. showed in A Theory for Neural Networks with Time Delays that temporal convolutions can be mathematically transformed into systems of ordinary differential equations. This allowed time-delay architectures to inherit the stability and convergence properties of standard additive neural networks.
Similarly, in reservoir computing, Christa Cuchiero et al. utilized Volterra series in Discrete-Time Signatures and Randomness in Reservoir Computing to quantify approximation errors when embedding fading-memory nonlinear systems into state-affine systems, providing a geometric explanation for why randomly initialized recurrent networks can effectively learn complex dynamics.
Principled Operator Design and Stochastic Modeling
Beyond analyzing existing networks, the Volterra framework allows engineers to design entirely new, principled architectures that bypass the limitations of traditional deep stacking.
Efficient Neural Processes
Traditional convolutional neural processes often require restrictive, rigid input grids to perform spatial convolutions, while attention-based alternatives suffer from quadratic computational scaling. In Revisiting Neural Processes via Fourier Transform and Volterra Series, Peiman Mohseni et al. leveraged Volterra series to represent complex nonlinear mappings as a sum of higher-order convolutions. By executing these convolutions directly in the frequency domain, the authors created a family of neural processes that achieve translation equivariance and global receptive fields with linear computational complexity, offering a mathematically transparent alternative to deep layer stacking.
Non-Linear Gaussian Processes
In probabilistic modeling, multi-output Gaussian processes (MOGPs) traditionally rely on linear process convolutions, which fail to capture nonlinear cross-dependencies between different observed outputs. Mauricio A Álvarez et al. resolved this in Non-linear process convolutions for multi-output Gaussian processes by introducing a Volterra series expansion to connect latent Gaussian processes to observed outputs. By deriving closed-form expressions for the exact mean and covariance of these non-linear transformations, the authors successfully captured complex, higher-order temporal dependencies while maintaining the rigorous framework of joint Gaussian processes.
Overcoming Parameter Explosion in Practice
If Volterra series are so powerful, why aren't they used everywhere? The primary challenge is the curse of dimensionality: as the non-linearity order increases, the number of coefficients in the multidimensional kernels explodes.
To make these models practical, researchers use structured approximations to compress the kernels:
- Orthonormal Basis Functions (OBFs): Instead of estimating every point in a multi-dimensional grid, the kernels can be projected onto a compact set of pre-defined mathematical functions. As explained by Gustavo Henrique Costa Oliveira et al. in An introduction to models based on Laguerre, Kautz and other related orthonormal functions - Part II: non-linear models, expanding Volterra kernels using OBFs provides a highly structured framework for system identification that mitigates over-parameterization.
- Fast Estimation Algorithms: For highly chaotic real-world signals like speech, Hemant A. Patil et al. demonstrated in Nonlinear prediction of speech signal using volterra-wiener series that the Volterra-Wiener-Korenberg (VWK) series approximation can be paired with fast coefficient estimation algorithms. This approach yields lower prediction errors and flatter residual spectra than traditional linear prediction methods by accurately capturing the physical nonlinearities of speech production.
Finally, understanding the boundaries of these nonlinear expansions helps clarify when simpler models are sufficient. In Best Linear Approximation of Nonlinear Continuous-Time Systems Subject to Process Noise and Operating in Feedback, Rik Pintelon et al. used Volterra series to formally define the class of fading-memory nonlinear systems that can be safely simplified via linear approximations, even when operating inside complex feedback loops plagued by internal process noise.
By treating deep neural networks and complex physical processes as expressions of a underlying Volterra series, the machine learning community can move away from pure empiricism. The framework provides the mathematical language necessary to dissect, simplify, and intentionally design the next generation of nonlinear sequence models.
Key Foundational Work
A Theory for Neural Networks with Time Delays (1990) — contributed 2 concept elements to this direction.
An introduction to models based on Laguerre, Kautz and other related orthonormal functions - Part II: non-linear models (2012) — contributed 1 concept elements to this direction.