Stochastic error whitening algorithm for linear filter estimation with noisy data

Yadunandana N. Rao, Deniz Erdogmus, Geetha Y. Rao, José Carlos PríncipeNeural Networks 2003 · DOI

TL;DR

This paper introduces a novel "Error Whitening Criterion" (EWC) algorithm for linear filter estimation, which significantly outperforms traditional Mean Squared Error (MSE) methods when dealing with noisy input data. By focusing on whitening the error signal at non-zero lags, EWC effectively mitigates the bias introduced by noise, leading to more accurate parameter estimates and improved performance in applications like inverse control.

What Problem Does This Paper Address?

Estimating parameters of linear filters is a fundamental problem in signal processing and control systems. The standard approach, Mean Squared Error (MSE) minimization, works well when the input data is clean. However, in real-world scenarios, input signals are often corrupted by noise. This additive noise biases the input covariance matrix, leading to inaccurate parameter estimates when using MSE-based methods. While techniques like Total Least Squares (TLS) exist to address this bias, they often have restrictive assumptions, such as requiring identical noise variances in input and desired signals. This paper tackles the challenge of accurately estimating linear filter parameters in the presence of additive white noise, where traditional MSE methods fail and other advanced techniques have limitations.

Methods

The core contribution is the development of a stochastic gradient algorithm based on the Error Whitening Criterion (EWC). Unlike MSE which minimizes the zero-lag autocorrelation of the error, EWC aims to minimize the error autocorrelation at non-zero lags. The paper derives an online stochastic gradient algorithm, termed EWC-LMS, by formulating a cost function that penalizes non-zero lag error autocorrelations. This criterion enforces a form of "whitening" on the error signal.

Key aspects of the method include:

  • EWC Cost Function: Defined as $J(w) = |E(\hat{e}_k^2) + \beta E(\hat{e}_k \hat{e}_{k-L}^2)|$, where $\beta$ is a parameter and $L$ is the lag. Setting $\beta = -0.5$ and minimizing this function leads to an unbiased estimate of the true weight vector.
  • EWC-LMS Algorithm: A recursive update rule derived from the EWC cost function. It's presented as: $w_{k+1} = w_k + \eta \text{sign}(\hat{e}_k^2 + \beta \hat{e}_k^2)(\hat{e}_k x_k + \beta \hat{e}_k x_k)$, where $\eta$ is the step-size. For $\beta = -0.5$, this reduces to a form similar to the well-known LMS algorithm but with a different error weighting.
  • Convergence Analysis: The paper provides theoretical convergence proofs for the EWC-LMS algorithm under both noise-free and noisy data conditions, establishing bounds on the step-size for convergence in the mean.
  • Algorithm Extensions: To improve robustness and convergence speed, the authors propose:
  • Normalization: Using the Hessian matrix's eigenvalue to normalize the step-size, leading to a normalized EWC-LMS.
  • Multiple Lags: Extending the EWC cost function to include constraints at multiple non-zero lags, enhancing robustness at the cost of increased computational complexity.

Datasets

The paper uses simulated data for its experiments. Specifically, it generates:

  • Noise-free input signals: Described as "sufficiently colored."
  • Additive white Gaussian noise: Added to the input signal.
  • Noise-free desired signals: Generated by passing the clean input through an unknown system.
  • An unknown system: Modeled as an all-pole system for inverse control experiments.

No specific named datasets are used; rather, the authors generate synthetic data to control signal-to-noise ratios (SNRs) and system parameters for evaluation.

Key Insights and Ideas

  • The power of non-zero lag error correlation: The fundamental insight is that minimizing error correlation at non-zero lags, rather than just at zero lag (as in MSE), can effectively remove the bias introduced by noisy inputs. This is a clever way to "whiten" the error signal.
  • EWC as a generalization of MSE: The EWC cost function naturally includes MSE as a special case (when $\beta=0$), highlighting its broader applicability.
  • Unbiased estimation in the presence of noise: The EWC-LMS algorithm demonstrably achieves unbiased parameter estimates even with significant additive white noise, a feat that MSE struggles with.
  • Robustness to noise variance: Unlike TLS, EWC does not require the noise variances in the input and desired signals to be identical, making it more practical for a wider range of applications.
  • Improved inverse control: The paper shows that using EWC for system identification leads to better-designed inverse controllers, which can more accurately track desired trajectories, especially under noisy conditions.

Key Results

  • Superior performance at low SNR: In system identification experiments, EWC-LMS significantly outperforms standard LMS at low Signal-to-Noise Ratios (SNRs) of -10 dB and 0 dB. For example, histograms of error vector norms show a much tighter distribution around zero for EWC-LMS compared to LMS.
  • Comparable performance at high SNR: At higher SNRs (e.g., 10 dB), EWC-LMS performance is comparable to LMS.
  • TLS advantage at high SNR: Batch-type numerical TLS methods achieve the best results at high SNRs, but their limitations at lower SNRs are evident.
  • Effective inverse controller: In the inverse control application, the controller designed using EWC-identified plant models exhibited superior tracking performance and significantly lower tracking errors compared to controllers based on MSE-identified models. The error histograms for EWC were concentrated around zero, indicating high accuracy.
  • Magnitude and phase response matching: The magnitude and phase responses of the EWC controller-model pair closely matched the desired transfer function, whereas the MSE-based pair showed significant deviations.

Limitations and Future Work

  • Computational Cost of Extensions: While extensions like using multiple lags improve robustness, they increase computational complexity. The paper notes that for $M$ lags, the complexity becomes $O(M^2)$.
  • Colored Noise and Non-linear Systems: The current EWC criterion is primarily designed for additive white noise. The authors suggest that future research will focus on extending the criterion to handle colored noise and non-linear system identification.
  • Further Analysis of Multi-Lag EWC: A more in-depth analysis of the multi-lag EWC algorithm (Eq. 34) is mentioned as being beyond the scope of the current paper and will be presented in a later publication.
  • Step-size selection: While convergence is proven under certain step-size bounds, selecting the optimal step-size for a desired tradeoff between convergence speed and asymptotic accuracy remains a non-trivial problem, as is common with stochastic gradient algorithms.

Summary updated

BibTeX
@article{rao2003stochastic,
  author = {Yadunandana N. Rao and Deniz Erdogmus and Geetha Y. Rao and José Carlos Príncipe},
  title = {{Stochastic error whitening algorithm for linear filter estimation with noisy data}},
  journal = {Neural Networks},
  year = {2003},
  doi = {10.1016/s0893-6080(03)00109-6},
}
Figure