Deep Learning Meets Adaptive Filtering: A Stein's Unbiased Risk Estimator Approach

Zahra Esmaeilbeig, Mojtaba Soltanalian

Abstract

This paper revisits two prominent adaptive filtering algorithms, namely recursive least squares (RLS) and equivariant adaptive source separation (EASI), through the lens of algorithm unrolling. Building upon the unrolling methodology, we introduce novel task-based deep learning frameworks, denoted as Deep RLS and Deep EASI. These architectures transform the iterations of the original algorithms into layers of a deep neural network, enabling efficient source signal estimation by leveraging a training process. To further enhance performance, we propose training these deep unrolled networks utilizing a surrogate loss function grounded on Stein's unbiased risk estimator (SURE). Our empirical evaluations demonstrate that the Deep RLS and Deep EASI networks outperform their underlying algorithms. Moreover, the efficacy of SURE-based training in comparison to conventional mean squared error loss is highlighted by numerical experiments. The unleashed potential of SURE-based training in this paper sets a benchmark for future employment of SURE either for training purposes or as an evaluation metric for generalization performance of neural networks.

Introduction

In signal processing, Blind Source Separation (BSS) remains a fundamental challenge: recovering independent source signals from a set of observed mixtures without prior knowledge of the mixing process. For decades, classical adaptive filters such as Recursive Least Squares (RLS) and Equivariant Adaptive Source Separation (EASI) have served as the primary tools for this task. However, these traditional methods are often limited by the need for meticulous manual tuning of hyperparameters—such as forgetting factors and step sizes—and frequently suffer from slow convergence rates.

While modern Deep Neural Networks (DNNs) offer a potential solution to these limitations, they are often criticized for being "black boxes" that lack interpretability and require massive datasets. This research bridges the gap between these two worlds by "unrolling" classical adaptive algorithms into deep architectures, combining the physical interpretability of signal processing with the data-driven power of deep learning.

Methodology: Algorithm Unrolling

The core innovation of this work lies in Algorithm Unrolling (or unfolding). This technique maps the iterative steps of a classical algorithm onto the layers of a deep neural network.

Deep RLS and Deep EASI

In the proposed Deep RLS and Deep EASI architectures, each iteration of the original algorithm is treated as a single layer in a deep network. Parameters that were historically fixed or manually tuned—such as the RLS forgetting factor ($\beta$) or the EASI step size ($\lambda$)—are transformed into learnable weights. This allows the model to optimize its internal logic based on the specific characteristics of the data it encounters.

Overparameterization and Nonlinearities

To further enhance the flexibility of these models, the authors replace fixed nonlinear functions within the algorithms with small, fully connected sub-networks. This overparameterization enables the network to "learn" the optimal nonlinearity for a given signal distribution, rather than relying on a pre-defined mathematical guess.

Unsupervised Training via SURE

A significant hurdle in training BSS models is the lack of "ground truth" (the original clean signals) in real-world scenarios. To address this, the authors utilize Stein’s Unbiased Risk Estimator (SURE). SURE provides a mathematical framework to estimate the Mean Squared Error (MSE) using only noisy observations. By using SURE as a surrogate loss function, the networks can be trained in an unsupervised or self-supervised manner, achieving performance levels comparable to models trained with full access to ground truth data.

Experimental Setup

The framework was validated using synthetically generated data to ensure mathematical rigor:

  • Source Signals: Generated from sub-Gaussian distributions (specifically i.i.d. uniform distributions $U(0, 1)$).
  • Mixing Matrices: Random matrices $A \sim \mathcal{N}(0, I)$ were generated for each sample to test the model's ability to generalize across diverse mixing scenarios.
  • Scale: The models were trained on $10^3$ samples and evaluated on a test set of $10^2$ samples.

Key Results and Discussion

The empirical results highlight several advantages of the deep unrolled approach:

  1. Accelerated Convergence: Deep EASI and Deep RLS reached low average MSE in approximately 50 iterations (layers). In comparison, the classical EASI and RLS algorithms required 100 and 250 iterations, respectively, to achieve similar performance. This represents a significant improvement in computational efficiency for real-time applications.
  2. Enhanced Accuracy: The "Deep" variants consistently achieved lower final MSE than their classical counterparts. This suggests that the learned step sizes and optimized nonlinearities are more effective than the static parameters used in traditional signal processing.
  3. SURE Efficacy: Numerical experiments confirmed that SURE-based training effectively tracks the performance of MSE-based training. This validates SURE as a powerful tool for training neural networks in environments where clean reference signals are unavailable.
  4. Structural Efficiency: By employing Recurrent Neural Network (RNN) logic—specifically sharing weights across layers—the models remain lightweight and capable of processing signal sequences of varying lengths without a massive increase in parameter count.

Conclusion and Future Work

This paper demonstrates that unrolling classical adaptive filters into deep networks provides a robust, interpretable, and high-performance alternative to both traditional filters and generic DNNs. The successful application of SURE for unsupervised training opens new doors for real-time signal processing in the field.

Future research will focus on applying these models to real-world datasets, such as biomedical EEG/ECG signals or complex audio environments. Additionally, while SURE was successfully implemented for Deep EASI, the authors aim to derive a more comprehensive analytical SURE loss for the Deep RLS architecture to further expand the framework's utility.

Summary updated

Abstract

This paper revisits two prominent adaptive filtering algorithms, namely recursive least squares (RLS) and equivariant adaptive source separation (EASI), through the lens of algorithm unrolling. Building upon the unrolling methodology, we introduce novel task-based deep learning frameworks, denoted as Deep RLS and Deep EASI. These architectures transform the iterations of the original algorithms into layers of a deep neural network, enabling efficient source signal estimation by leveraging a training process. To further enhance performance, we propose training these deep unrolled networks utilizing a surrogate loss function grounded on Stein's unbiased risk estimator (SURE). Our empirical evaluations demonstrate that the Deep RLS and Deep EASI networks outperform their underlying algorithms. Moreover, the efficacy of SURE-based training in comparison to conventional mean squared error loss is highlighted by numerical experiments. The unleashed potential of SURE-based training in this paper sets a benchmark for future employment of SURE either for training purposes or as an evaluation metric for generalization performance of neural networks.
BibTeX
@inproceedings{esmaeilbeig2023deep,
  author = {Zahra Esmaeilbeig and Mojtaba Soltanalian},
  title = {{Deep Learning Meets Adaptive Filtering: A Stein's Unbiased Risk Estimator Approach}},
  booktitle = {Allerton},
  year = {2023},
  url = {https://openreview.net/forum?id=v1sIgolRFK},
}
Figure