Multi-innovation-based online variable projection algorithm for a class of nonlinear models

Jian-Nan Su, Xiang-Xiang Su, Guangyong Chen, Min Gan, C. L. Philip Chen

Abstract

Separable nonlinear models (SNMs) are powerful tools for system identification, data analysis, and machine learning, thanks to their flexible structure and remarkable ability to capture nonlinear behaviors. However, compared with the offline one, the online identification of SNMs is more challenging, due to the dynamic nature of the nonlinear systems and the interdependence of the different parts of parameters in SNMs. Existing methods, such as the recursive Gauss-Newton (RGN) and recursive variable projection (RVP) methods, which cannot effectively handle the parameter interdependence, often suffer from slow convergence and poor performance in this scenario. In this paper, we propose a novel multi-innovation-based recursive variable projection (MIRVP) algorithm, which extends the RVP algorithm with a multi-innovation strategy. This strategy enables the algorithm to handle the interaction of the linear and nonlinear parameters more effectively, by using multiple past innovations instead of only the current one, thus improving the identification effect. We demonstrate the effectiveness of our MIRVP algorithm on both synthetic and real world datasets, showing its advantages in convergence speed and robustness over existing methods.

Introduction

Separable Nonlinear Models (SNMs) are widely utilized across signal processing, industrial control, and machine learning because they allow for a structural decomposition into linear and nonlinear parameter sets. While "offline" identification methods—where the entire dataset is available—are well-established, "online" identification remains a significant hurdle. In real-time streaming environments, the dynamic nature of data and the tight coupling between linear and nonlinear parameters often lead to poor performance in traditional recursive algorithms.

The primary challenge is parameter interdependence. In an online setting, updates to nonlinear parameters immediately shift the optimal values for linear parameters. Conventional methods like Recursive Gauss-Newton (RGN) often fail to account for this interaction with sufficient precision, resulting in slow convergence and an inability to track rapidly changing systems.

The MIRVP Algorithm

To address these limitations, this research introduces the Multi-innovation-based Recursive Variable Projection (MIRVP) algorithm. This approach integrates two sophisticated mathematical strategies:

  1. Variable Projection (VP): This technique reduces the complexity of the optimization problem by mathematically "eliminating" linear parameters, expressing them as functions of the nonlinear ones.
  2. Multi-innovation (MI) Strategy: Unlike standard recursive algorithms that update estimates based solely on the most recent error (innovation), the MI strategy utilizes a "moving window" of the $v$ most recent innovations. This provides the algorithm with a short-term memory that stabilizes updates and improves accuracy.

Implementation Steps

The MIRVP algorithm operates in a three-step cycle for every incoming data point, ensuring that both parameter sets are refined in tandem:

  • Step 1: Linear Parameter Elimination. The multi-innovation window is used to estimate the linear parameters, effectively removing them from the immediate nonlinear optimization task.
  • Step 2: Nonlinear Parameter Update. The nonlinear parameters are updated using a reduced objective function. The paper details two variants: GN-MIRVP (a second-order Gauss-Newton approach) and G-MIRVP (a first-order gradient-based approach for high-dimensional problems).
  • Step 3: Linear Parameter Refinement. Finally, the linear parameters are updated using the Recursive Least Squares (RLS) method, conditioned on the newly updated nonlinear estimates.
The flowchart of the MIRVP algorithm illustrating the three-step recursive identification process.

Experimental Validation

The researchers benchmarked MIRVP against traditional methods across several diverse datasets to test its robustness and speed.

Synthetic and Real-World Benchmarks

In simulations using a complex Synthetic Exponential Model, the GN-MIRVP variant demonstrated a dramatic improvement in convergence speed compared to RGN. A key finding was the impact of the innovation length ($v$); as $v$ increased from 1 to 6, the algorithm reached lower error thresholds significantly faster, highlighting the power of the multi-innovation memory.

Performance comparison of second-order online learning algorithms for the exponential model, showing the impact of innovation length (v).

For real-world applications, the algorithm was tested on the Box-Jenkins Gas Furnace dataset and the Glass Tube Drawing Process. In both cases, MIRVP provided superior one-step-ahead predictions, proving its capability to handle the noise and non-stationarity inherent in industrial data.

Machine Learning Application

The algorithm's versatility was further proven on the MNIST Handwritten Digits dataset. When used to train feedforward neural networks, MIRVP achieved an average prediction accuracy of over 85%. This significantly outperformed standard Stochastic Gradient Descent (SGD), which hovered around 60%, and an alternating SGD-RLS approach at 78%.

Conclusion

The MIRVP algorithm represents a significant advancement in the online identification of separable nonlinear models. By leveraging multi-innovation theory, it effectively manages the interdependence of parameters that cripples simpler recursive methods. While increasing the innovation length ($v$) introduces a higher computational cost per iteration, the gains in convergence speed and steady-state accuracy make it a superior choice for high-precision real-time applications. Future work will focus on automating the selection of the optimal innovation window and extending the framework to non-separable nonlinear architectures.

Summary updated

Abstract

<title>Abstract</title> Separable nonlinear models (SNMs) are powerful tools for system identification, data analysis, and machine learning, thanks to their flexible structure and remarkable ability to capture nonlinear behaviors. However, compared with the offline one, the online identification of SNMs is more challenging, due to the dynamic nature of the nonlinear systems and the interdependence of the different parts of parameters in SNMs. Existing methods, such as the recursive Gauss-Newton (RGN) and recursive variable projection (RVP) methods, which cannot effectively handle the parameter interdependence, often suffer from slow convergence and poor performance in this scenario. In this paper, we propose a novel multi-innovation-based recursive variable projection (MIRVP) algorithm, which extends the RVP algorithm with a multi-innovation strategy. This strategy enables the algorithm to handle the interaction of the linear and nonlinear parameters more effectively, by using multiple past innovations instead of only the current one, thus improving the identification effect. We demonstrate the effectiveness of our MIRVP algorithm on both synthetic and real world datasets, showing its advantages in convergence speed and robustness over existing methods.
BibTeX
@inproceedings{su2024multi,
  author = {Jian-Nan Su and Xiang-Xiang Su and Guangyong Chen and Min Gan and C. L. Philip Chen},
  title = {{Multi-innovation-based online variable projection algorithm for a class of nonlinear models}},
  booktitle = {Nonlinear},
  year = {2024},
  doi = {10.1007/s11071-024-09799-y},
}
Figure