Mitigating Parameter Explosion in Nonlinear System Identification

18 concepts14 papers

When modeling a complex dynamical system—such as a high-fidelity loudspeaker, a wireless power amplifier, or a chemical reactor—engineers must capture two distinct phenomena simultaneously: nonlinearity (where the output is not strictly proportional to the input) and memory (where the current output depends on past inputs).

For decades, the mathematical gold standard for this task has been the Volterra series. Think of a Volterra series as a Taylor series with memory. A first-order Volterra kernel represents a standard linear filter. A second-order kernel captures pairs of past inputs interacting quadratically, a third-order kernel captures triplets, and so on.

$$\begin{aligned} y(t) = h_0 &+ \sum_{\tau_1=0}^{M-1} h_1(\tau_1) x(t-\tau_1) \\ &+ \sum_{\tau_1=0}^{M-1} \sum_{\tau_2=0}^{M-1} h_2(\tau_1, \tau_2) x(t-\tau_1)x(t-\tau_2) + \dots \end{aligned}$$

While elegant, this formulation harbors a fatal flaw: the curse of dimensionality. As noted by Anna Wigren et al. in their overview of nonlinear system identification, the complexity of these models scales exponentially. If a system has a memory length of $M$ samples and we model up to a nonlinearity degree of $P$, the number of coefficients required for the $P$-th order kernel scales as $M^P$. For a modest memory of $M=100$ and a fifth-order nonlinearity ($P=5$), a single kernel demands $100^5 = 10$ billion parameters.

This exponential parameter explosion makes traditional estimation techniques like least squares computationally impossible and data-starved. To make nonlinear system identification practical for real-time applications, a shared research direction has emerged: compressing this high-dimensional parameter space by exploiting structural redundancies.


Tensor Networks: Factoring the Grid

The most direct way to defeat the curse of dimensionality is to recognize that a high-order Volterra kernel is simply a multi-dimensional grid—a tensor. In physical systems, the entries in this grid are rarely independent; they possess low-rank structural relationships.

Instead of storing every individual element of an $P$-dimensional tensor, we can factorize it into a chain of low-dimensional, interconnected tensors. This is known as a Tensor Network (TN) or Tensor Train (TT) decomposition.

Standard Tensor (Explosive):  [ M × M × M × ... × M ]  --> M^P parameters

Tensor Train (Linear):       [M×R] - [R×M×R] - [R×M×R] - ...

- [R×M] --> P × M × R^2 parameters

By representing the kernel as a Tensor Train with a maximum internal rank $R$, the parameter count plummets from $M^P$ to roughly $P \cdot M \cdot R^2$. The exponential scaling is completely replaced by linear scaling.

In Volterra kernels of bilinear systems have tensor train structure, P.M.S. Burt proved that for bilinear and linear-analytic systems, this Tensor Train structure is not just a good approximation—it is mathematically exact. By leveraging this insight, P.M.S. Burt designed a "Cascade-TT" architecture that slashes real-time computational overhead without losing any modeling accuracy.

To identify these compressed models from data, researchers had to redesign classical estimation algorithms to operate entirely within the tensor-network space. kim batselier et al. developed the Tensor Network Kalman Filter (TNKF), an algorithm capable of recursively updating state vectors of astronomical sizes (e.g., $10^9$ elements) by keeping the covariance matrices and state vectors strictly in a compressed tensor format. This concept was subsequently expanded to Multiple-Input-Multiple-Output (MIMO) systems by Kim Batselier et al. and Kim Batselier et al., who introduced alternating linear schemes (ALS) to optimize high-order MIMO Volterra coefficients in seconds. Similar tensor frameworks have been successfully adopted in electronic design automation by Zheng Zhang et al. to simulate complex analog and radio-frequency components without encountering storage bottlenecks.


Orthonormal Basis Functions: Smoothing the Memory

Another way to bypass the parameter explosion is to change how we represent time. Standard Volterra models use discrete time delays ($x(t-1), x(t-2), \dots$), treating every past time step as an independent dimension. This ignores the fact that physical memory usually decays smoothly over time.

By projecting the input history onto a set of Orthonormal Basis Functions (OBFs)—such as Laguerre or Kautz functions—we can compress the memory dimension $M$ before it ever interacts with the nonlinearity. These basis functions come pre-equipped with built-in exponential or resonant decay profiles that mirror physical systems.

Discrete Delays:  x(t-1), x(t-2), x(t-3), ... x(t-100)  --> 100 dimensions
OBF Projection:   [Input History] ──> [ 3 Kautz Filters ] ──> 3 dimensions

Alex da Rosa et al. demonstrated this in An optimal expansion of Volterra models using independent Kautz bases for each kernel dimension, showing that assigning independent Kautz bases to each dimension of a Volterra kernel yields highly parsimonious representations of complex dynamics. In a broader review, Gustavo Henrique Costa Oliveira et al. established a unified framework for non-linear OBF models, illustrating how these bases prevent over-parameterization and eliminate the hypersensitivity to model order that plagues traditional nonlinear finite impulse response (NFIR) structures.


Compressive Sensing: Dropping the Zeros

In many real-world applications, even if a high-order interaction space is technically vast, only a tiny fraction of those interactions actually influence the output. The underlying Volterra kernel is inherently sparse.

Instead of collecting massive datasets to solve an over-determined system of equations, we can turn to compressive sensing. Vassilis Kekatos et al. applied compressive sampling principles to Volterra and polynomial regression models in Sparse Volterra and Polynomial Regression Models: Recoverability and Estimation. By proving that sparse nonlinear systems satisfy the Restricted Isometry Property (RIP) bounds, the authors demonstrated that high-order systems can be perfectly recovered using far fewer measurements than classical sampling theorems demand.

Complementing this data-side sparsity, P.M.S. Burt focused on algorithmic sparsity in Efficient Computation of Bilinear Approximations and Volterra Models of Nonlinear Systems. By combining non-redundant Taylor expansions with Carleman bilinearization, P.M.S. Burt eliminated redundant calculations to achieve speedups exceeding 300x, proving that structural pruning can make kernel derivation highly practical.


Deep Learning: Implicit Composition

A final, increasingly prominent strategy abandons explicit polynomial expansions altogether. Rather than mapping out every high-order interaction term by hand, we can use deep neural networks to compose nonlinearities implicitly through hierarchical layers.

Volterra:       Input ──> [Explicit Combinatorial Combinations] ──> Output (Explosive)
Deep Network:   Input ──> [Layer 1] ──> [Layer 2] ──> [Layer 3] ──> Output (Compact)

Carl R. Andersson et al. bridged the gap between control engineering and deep learning in Deep Convolution Networks in System Identification, showing that Temporal Convolutional Networks (TCNs) naturally scale to long-memory systems without encountering the parameter explosion that cripples Volterra series.

Other architectures use specialized mathematical functions inside the network to accelerate convergence. For instance, Wang Li et al. introduced an Improved Elman Neural Network (IENN) that swaps standard activation functions for Chebyshev orthogonal polynomials, capturing circuit memory effects with minimal parameter overhead. Similarly, Jin Xiangjun et al. proposed a Deep Reconstruction Model (DRM) that uses Restricted Boltzmann Machines for unsupervised weight initialization, providing a highly compact alternative to traditional mathematical frameworks for real-time engineering applications.

Key Foundational Work

Sparse Volterra and Polynomial Regression Models: Recoverability and Estimation (2011) — contributed 1 concept elements to this direction.
An optimal expansion of Volterra models using independent Kautz bases for each kernel dimension (2008) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_curse_of_dimensionality_high_order_model_2ec8db08,
  title = {curse of dimensionality (high-order model parameter explosion)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/curse-of-dimensionality-high-order-model-parameter-explosion/rdr_2ec8db08830f462fa96bcf9199f2f962},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_curse_of_dimensionality_high_order_model_2ec8db08_v1,
  title = {curse of dimensionality (high-order model parameter explosion)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/curse-of-dimensionality-high-order-model-parameter-explosion/rdr_2ec8db08830f462fa96bcf9199f2f962/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake