Multi-innovation-based online variable projection algorithm for a class of nonlinear models

Jian-Nan Su, Xiang-Xiang Su, Guangyong Chen, Min Gan, C. L. Philip Chen

TL;DR

The paper introduces MIRVP, a novel online algorithm designed to identify parameters in separable nonlinear models (SNMs) in real-time. By combining the "variable projection" strategy with a "multi-innovation" approach—which uses a window of past errors rather than just the latest one—the algorithm achieves significantly faster convergence and higher accuracy than traditional recursive methods.

What Problem Does This Paper Address?

Many real-world systems, from signal processing to industrial manufacturing, are modeled using Separable Nonlinear Models (SNMs). These models are unique because their parameters can be split into two groups: a linear part and a nonlinear part. While we have robust "offline" tools to solve these when all data is available at once, doing it "online" (as data streams in) is notoriously difficult.

The core challenge is parameter interdependence. In an online setting, the linear and nonlinear parameters are tightly coupled; a small change in one affects the optimal value of the other. Existing methods like Recursive Gauss-Newton (RGN) often treat these parameters as independent or fail to account for their complex interaction, leading to slow learning speeds and poor tracking of dynamic systems.

Methods

The authors propose the Multi-innovation-based Recursive Variable Projection (MIRVP) algorithm. It builds on two key concepts:

  1. Variable Projection (VP): This technique mathematically "eliminates" the linear parameters by expressing them as functions of the nonlinear ones. This reduces the complexity of the optimization problem.
  2. Multi-innovation (MI) Strategy: Standard recursive algorithms update their estimates based only on the current error (innovation). The MI strategy instead uses a "moving window" of the $v$ most recent innovations.

The MIRVP process follows a three-step cycle for every new data point:

  • Step 1: Use the multi-innovation window to estimate (and effectively eliminate) the linear parameters.
  • Step 2: Update the nonlinear parameters using a reduced objective function. The authors provide two versions: GN-MIRVP (using a second-order Gauss-Newton approach) and G-MIRVP (a first-order gradient-based approach for larger-scale problems).
  • Step 3: Refine the linear parameters using Recursive Least Squares (RLS) based on the updated nonlinear estimates.

The flowchart for the gradient-based version is shown below:

The flowchart of the gradient-based MIRVP algorithm

Datasets

The researchers validated MIRVP across a diverse range of benchmarks:

  • Synthetic Exponential Model: A complex three-term exponential function used to test convergence speed.
  • Box-Jenkins Gas Furnace: A classic system identification dataset recording CO2 concentration against gas flow rates.
  • Glass Tube Drawing Process: Real-world industrial data used to model the dynamic behavior of glass wall thickness.
  • MNIST Handwritten Digits: A standard machine learning dataset used to demonstrate MIRVP's effectiveness in training feedforward neural networks.

Key Insights and Ideas

  • Memory Matters: The "aha moment" here is that using past innovations ($v > 1$) provides a stabilizing "memory" for the algorithm. This allows it to handle the parameter coupling much more effectively than "single-innovation" methods.
  • The Power of $v$: There is a clear performance boost as the innovation length ($v$) increases. For example, in the exponential model, moving from $v=1$ to $v=6$ dramatically improved the convergence rate.
  • Theoretical Soundness: Beyond empirical results, the authors provide a mathematical proof that the estimation errors of MIRVP are mean-square bounded, ensuring the algorithm's stability.

Key Results

The MIRVP algorithm consistently outperformed standard methods like Stochastic Gradient Descent (SGD) and Recursive Gauss-Newton (RGN) across all tests.

  • Speed and Accuracy: In the exponential model simulations, GN-MIRVP reached lower error levels significantly faster than its competitors (see Figure 2).
  • Neural Network Training: On the MNIST dataset, MIRVP achieved an average prediction accuracy of over 85%, compared to roughly 60% for standard SGD and 78% for an alternating SGD-RLS approach.
  • Real-world Robustness: For the gas furnace and glass tube datasets, MIRVP showed superior one-step-ahead prediction capabilities, proving it can handle the noise and non-stationarity of real-world data.
Performance comparison of second-order online learning algorithms for the exponential model

Limitations and Future Work

While MIRVP is powerful, it does come with trade-offs:

  • Computational Cost: Increasing the innovation length ($v$) improves accuracy but also increases the computational burden per iteration. Finding the "sweet spot" for $v$ is currently a manual process.
  • Hyperparameter Tuning: Like many recursive algorithms, performance depends on the initial choice of step sizes and covariance matrices.
  • Generalization: The current work is tailored to separable nonlinear models. Future research could explore how these multi-innovation strategies might be adapted for more general, non-separable nonlinear architectures.

Summary updated

Abstract

<title>Abstract</title> Separable nonlinear models (SNMs) are powerful tools for system identification, data analysis, and machine learning, thanks to their flexible structure and remarkable ability to capture nonlinear behaviors. However, compared with the offline one, the online identification of SNMs is more challenging, due to the dynamic nature of the nonlinear systems and the interdependence of the different parts of parameters in SNMs. Existing methods, such as the recursive Gauss-Newton (RGN) and recursive variable projection (RVP) methods, which cannot effectively handle the parameter interdependence, often suffer from slow convergence and poor performance in this scenario. In this paper, we propose a novel multi-innovation-based recursive variable projection (MIRVP) algorithm, which extends the RVP algorithm with a multi-innovation strategy. This strategy enables the algorithm to handle the interaction of the linear and nonlinear parameters more effectively, by using multiple past innovations instead of only the current one, thus improving the identification effect. We demonstrate the effectiveness of our MIRVP algorithm on both synthetic and real world datasets, showing its advantages in convergence speed and robustness over existing methods.
BibTeX
@inproceedings{su2024multi,
  author = {Jian-Nan Su and Xiang-Xiang Su and Guangyong Chen and Min Gan and C. L. Philip Chen},
  title = {{Multi-innovation-based online variable projection algorithm for a class of nonlinear models}},
  booktitle = {Nonlinear},
  year = {2024},
  doi = {10.1007/s11071-024-09799-y},
}
Figure