Deep Learning Meets Adaptive Filtering: A Stein's Unbiased Risk Estimator Approach

Zahra Esmaeilbeig, Mojtaba Soltanalian

TL;DR

This paper bridges the gap between classical signal processing and modern AI by "unrolling" two iconic adaptive filtering algorithms—Recursive Least Squares (RLS) and Equivariant Adaptive Source Separation (EASI)—into deep neural network architectures. By utilizing Stein’s Unbiased Risk Estimator (SURE) as a surrogate loss function, the authors enable these networks to outperform their classical counterparts in convergence speed and accuracy, even in unsupervised settings where ground truth is unavailable.

What Problem Does This Paper Address?

In the world of signal processing, Blind Source Separation (BSS) is a classic challenge: how do you recover individual source signals (like different voices in a room) from a mixture when you don't know the mixing process?

Classical adaptive filters like RLS and EASI have been the workhorses for this task for decades. However, they suffer from two main drawbacks:

  1. Manual Tuning: Their performance depends heavily on choosing the right hyperparameters (like forgetting factors or step sizes) and specific nonlinear functions.
  2. Slow Convergence: They often require many iterations to "settle" on a solution.

While generic Deep Neural Networks (DNNs) can solve these problems, they are often "black boxes" that require massive amounts of data and lack interpretability. This paper seeks a middle ground: an architecture that is as interpretable as a classical filter but as powerful and data-driven as a DNN.

Methods

The authors employ a technique called Algorithm Unrolling (or unfolding).

  • Deep RLS & Deep EASI: Instead of running a classical algorithm for $T$ iterations, they treat each iteration as a layer in a deep network. The parameters that were previously fixed (like the forgetting factor $\beta$ in RLS or the step size $\lambda$ in EASI) become learnable weights.
  • Overparameterization: They replace fixed nonlinearities with small, fully connected sub-networks. This allows the model to "learn" the optimal nonlinearity for the specific signal distribution it is processing.
  • SURE-based Training: This is the "secret sauce." Usually, training a network requires "ground truth" (the original, clean signals). In many real-world scenarios, we don't have that. The authors use Stein’s Unbiased Risk Estimator (SURE), a mathematical tool that provides an unbiased estimate of the Mean Squared Error (MSE) using only the noisy observations. This allows for unsupervised or self-supervised training.

Datasets

The study primarily uses synthetically generated data to validate the mathematical framework:

  • Source Signals: Generated from sub-Gaussian distributions (specifically i.i.d. uniform distributions $U(0, 1)$).
  • Mixing Matrices: Randomly generated matrices $A \sim \mathcal{N}(0, I)$ for each sample to ensure the model generalizes across different mixing scenarios.
  • Scale: Training sets of $10^3$ samples and test sets of $10^2$ samples.

Key Insights and Ideas

  • Interpretability is Built-In: Unlike a standard ResNet or Transformer, every layer in Deep RLS/EASI has a specific physical meaning derived from adaptive filter theory. This makes the models easier to debug and trust.
  • Efficiency through Parameter Sharing: By using Recurrent Neural Network (RNN) logic—sharing weights across "iterations"—the models remain lightweight and can handle signal sequences of varying lengths.
  • The Power of SURE: The paper demonstrates that SURE isn't just a theoretical curiosity; it's a practical surrogate for MSE. Training with SURE loss yielded results that were nearly indistinguishable from (and sometimes better than) training with full knowledge of the ground truth.
  • Hybrid Nature: The approach combines the "physics" of signal processing (the RLS/EASI update rules) with the "learning" of AI (backpropagation), leading to a "best of both worlds" scenario.

Key Results

  • Faster Convergence: In numerical simulations, Deep EASI and Deep RLS achieved very low average MSE in as few as 50 iterations/layers. In contrast, the classical EASI and RLS algorithms required roughly 100 and 250 iterations, respectively, to reach the same performance level.
  • Superior Accuracy: The "Deep" versions consistently outperformed their underlying classical algorithms in terms of final MSE, proving that the learned step sizes and nonlinearities are superior to manually tuned ones.
  • SURE vs. MSE: Figure 2 in the paper shows that SURE-based training effectively tracks the performance of MSE-based training, confirming its efficacy for unsupervised real-time signal processing.

Limitations and Future Work

  • Synthetic Focus: The current evaluation is limited to synthetic, uniformly distributed signals. Testing on real-world audio or biomedical signals (like EEG/ECG) is a necessary next step.
  • Mathematical Derivations: While the authors successfully implemented SURE for Deep EASI, they noted that the analytical derivation of the SURE loss for Deep RLS is complex and left for "future extension."
  • Computational Overhead: While unrolled networks use fewer parameters than generic DNNs, they still require a training phase, which might be a hurdle for ultra-low-power embedded systems compared to "plug-and-play" classical filters.

Summary updated

Abstract

This paper revisits two prominent adaptive filtering algorithms, namely recursive least squares (RLS) and equivariant adaptive source separation (EASI), through the lens of algorithm unrolling. Building upon the unrolling methodology, we introduce novel task-based deep learning frameworks, denoted as Deep RLS and Deep EASI. These architectures transform the iterations of the original algorithms into layers of a deep neural network, enabling efficient source signal estimation by leveraging a training process. To further enhance performance, we propose training these deep unrolled networks utilizing a surrogate loss function grounded on Stein’s unbiased risk estimator (SURE). Our empirical evaluations demonstrate that the Deep RLS and Deep EASI networks outperform their underlying algorithms. Moreover, the efficacy of SURE-based training in comparison to conventional mean squared error loss is highlighted by numerical experiments. The unleashed potential of SURE-based training in this paper sets a benchmark for future employment of SURE either for training purposes or as an evaluation metric for generalization performance of neural networks.
BibTeX
@inproceedings{esmaeilbeig2023deep,
  author = {Zahra Esmaeilbeig and Mojtaba Soltanalian},
  title = {{Deep Learning Meets Adaptive Filtering: A Stein's Unbiased Risk Estimator Approach}},
  booktitle = {Allerton},
  year = {2023},
  url = {https://openreview.net/forum?id=v1sIgolRFK},
}

Loading related content...

Figure