TL;DR
The paper introduces MIRVP, a novel online algorithm designed to identify parameters in separable nonlinear models (SNMs) in real-time. By combining the "variable projection" strategy with a "multi-innovation" approach—which uses a window of past errors rather than just the latest one—the algorithm achieves significantly faster convergence and higher accuracy than traditional recursive methods.
What Problem Does This Paper Address?
Many real-world systems, from signal processing to industrial manufacturing, are modeled using Separable Nonlinear Models (SNMs). These models are unique because their parameters can be split into two groups: a linear part and a nonlinear part. While we have robust "offline" tools to solve these when all data is available at once, doing it "online" (as data streams in) is notoriously difficult.
The core challenge is parameter interdependence. In an online setting, the linear and nonlinear parameters are tightly coupled; a small change in one affects the optimal value of the other. Existing methods like Recursive Gauss-Newton (RGN) often treat these parameters as independent or fail to account for their complex interaction, leading to slow learning speeds and poor tracking of dynamic systems.
Methods
The authors propose the Multi-innovation-based Recursive Variable Projection (MIRVP) algorithm. It builds on two key concepts:
- Variable Projection (VP): This technique mathematically "eliminates" the linear parameters by expressing them as functions of the nonlinear ones. This reduces the complexity of the optimization problem.
- Multi-innovation (MI) Strategy: Standard recursive algorithms update their estimates based only on the current error (innovation). The MI strategy instead uses a "moving window" of the $v$ most recent innovations.
The MIRVP process follows a three-step cycle for every new data point:
- Step 1: Use the multi-innovation window to estimate (and effectively eliminate) the linear parameters.
- Step 2: Update the nonlinear parameters using a reduced objective function. The authors provide two versions: GN-MIRVP (using a second-order Gauss-Newton approach) and G-MIRVP (a first-order gradient-based approach for larger-scale problems).
- Step 3: Refine the linear parameters using Recursive Least Squares (RLS) based on the updated nonlinear estimates.
The flowchart for the gradient-based version is shown below:
Datasets
The researchers validated MIRVP across a diverse range of benchmarks:
- Synthetic Exponential Model: A complex three-term exponential function used to test convergence speed.
- Box-Jenkins Gas Furnace: A classic system identification dataset recording CO2 concentration against gas flow rates.
- Glass Tube Drawing Process: Real-world industrial data used to model the dynamic behavior of glass wall thickness.
- MNIST Handwritten Digits: A standard machine learning dataset used to demonstrate MIRVP's effectiveness in training feedforward neural networks.
Key Insights and Ideas
- Memory Matters: The "aha moment" here is that using past innovations ($v > 1$) provides a stabilizing "memory" for the algorithm. This allows it to handle the parameter coupling much more effectively than "single-innovation" methods.
- The Power of $v$: There is a clear performance boost as the innovation length ($v$) increases. For example, in the exponential model, moving from $v=1$ to $v=6$ dramatically improved the convergence rate.
- Theoretical Soundness: Beyond empirical results, the authors provide a mathematical proof that the estimation errors of MIRVP are mean-square bounded, ensuring the algorithm's stability.
Key Results
The MIRVP algorithm consistently outperformed standard methods like Stochastic Gradient Descent (SGD) and Recursive Gauss-Newton (RGN) across all tests.
- Speed and Accuracy: In the exponential model simulations, GN-MIRVP reached lower error levels significantly faster than its competitors (see Figure 2).
- Neural Network Training: On the MNIST dataset, MIRVP achieved an average prediction accuracy of over 85%, compared to roughly 60% for standard SGD and 78% for an alternating SGD-RLS approach.
- Real-world Robustness: For the gas furnace and glass tube datasets, MIRVP showed superior one-step-ahead prediction capabilities, proving it can handle the noise and non-stationarity of real-world data.
Limitations and Future Work
While MIRVP is powerful, it does come with trade-offs:
- Computational Cost: Increasing the innovation length ($v$) improves accuracy but also increases the computational burden per iteration. Finding the "sweet spot" for $v$ is currently a manual process.
- Hyperparameter Tuning: Like many recursive algorithms, performance depends on the initial choice of step sizes and covariance matrices.
- Generalization: The current work is tailored to separable nonlinear models. Future research could explore how these multi-innovation strategies might be adapted for more general, non-separable nonlinear architectures.
Summary updated