A Comparison of Discrete-Time Operator Models for Nonlinear System Identification

NeurIPS 1994

TL;DR

This paper introduces a generalized "pi" operator for nonlinear system identification, unifying existing delta, gamma, and rho operators. By comparing these operators within neural network models, the authors demonstrate that more complex operators offer better performance but can introduce stability challenges, highlighting a trade-off between model complexity and robustness.

What Problem Does This Paper Address?

Identifying and modeling nonlinear systems from time-series data is a fundamental challenge in many engineering fields. Traditional methods using the "shift" operator (representing a time delay) can struggle with systems exhibiting lightly-damped, low-frequency dynamics, especially as sampling rates increase, leading to numerical instability and noise sensitivity. To address this, researchers have developed alternative discrete-time operators like the "delta" and "gamma" operators, which offer improved numerical properties. More recently, the "rho" operator was introduced, showing promise in adaptive control. However, a unified framework and a direct comparison of these operators for nonlinear system identification using neural networks were lacking. This paper aims to fill that gap by presenting a generalized operator and systematically comparing the performance of various operators in a neural network context.

Methods

The core of the paper lies in introducing a novel, generalized "pi" operator, defined by a bilinear transformation:

$$ \pi = \frac{\Delta (c_1 q - c_2)}{\Delta (c_3 q + c_4)} $$

This "pi" operator is shown to encompass the shift, delta, gamma, and rho operators by appropriately setting its coefficients ($c_1, c_2, c_3, c_4$). The authors then integrate these operators into the input stage of feedforward neural networks, creating "operator multilayer perceptron" (MLP($\nu$)) models. They propose a Gauss-Newton algorithm for updating both the neural network weights and the operator parameters simultaneously. The paper focuses on comparing the performance of models using these different operators for system identification tasks.

Datasets

The paper does not introduce new datasets. Instead, it uses two well-defined linear system models for evaluation:

  1. Experiment 1: A high bit rate echo cancellation system described by a rational transfer function with poles near the unit circle, representing a lightly-damped, low-frequency (LDLF) system.
  2. Experiment 2: A 3rd order lowpass filter, also tested in previous literature.

The input signal for both experiments was uniform white noise with unit variance.

Key Insights and Ideas

  • Operator Unification: The "pi" operator provides a powerful generalization, unifying the previously disparate delta, gamma, and rho operators under a single framework. This allows for a systematic exploration of operator properties.
  • Complexity vs. Performance Trade-off: More complex operators (rho and pi) generally yield better system identification performance (higher SNR) compared to simpler ones (shift and delta). This suggests that the increased degrees of freedom in these operators allow for more accurate modeling.
  • Stability Concerns with Complexity: The improved performance of complex operators comes at the cost of potential convergence and stability issues. The pi operator, despite its superior performance, was observed to be more prone to instability.
  • Delta Operator's Poor Performance: The delta operator, while designed to improve numerical properties, performed poorly in these nonlinear system identification tasks, exhibiting wide result variations.
  • Gamma and Rho Stability: The gamma and rho operators demonstrated good stability properties, making them more robust in practice, even if not always achieving the absolute best performance.

Key Results

  • Experiment 1 (LDLF System): The pi operator achieved the highest average Signal-to-Noise Ratio (SNR) of +10.0 dB, followed closely by the rho operator (+9.7 dB) and gamma operator (+5.7 dB). The shift operator performed poorly (+2.7 dB), and the delta operator was significantly worse (-7.1 dB).
  • Experiment 2 (Lowpass Filter): The pi operator again showed the best average SNR (+14.0 dB), with the rho (+13.3 dB) and gamma (+13.5 dB) operators also performing well. The shift operator achieved a respectable +10.7 dB, while the delta operator performed poorly (-21.5 dB).
  • Qualitative Findings: The pi operator consistently provided the best results but required stability correction mechanisms. The rho and gamma operators offered a good balance of performance and stability. The delta operator was found to be unreliable.

Limitations and Future Work

  • Linear Case Focus: The paper primarily focuses on linear system identification examples to simplify the analysis and highlight the operator's impact. The authors acknowledge that the derivations are applicable to multilayer perceptrons, but the full nonlinear case is not explored in depth.
  • Stability Mechanisms: The authors note the need for further work on mechanisms to maintain stability while retaining good convergence properties, especially for the more complex operators like pi.
  • Convergence Analysis: Future work should involve analyzing the convergence of the algorithms to design methods that ensure stability without hindering convergence.
  • Broader Task Range: The authors suggest that further investigation into the performance of these models on a wider range of tasks is warranted to fully understand their capabilities.

Summary updated

BibTeX
@inproceedings{1994comparison,
  title = {{A Comparison of Discrete-Time Operator Models for Nonlinear System Identification}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1994},
}

Loading related content...

Figure