Traditional adaptive filters, such as Least Mean Squares (LMS) or Recursive Least Squares (RLS), are built on the foundation of the Mean Square Error (MSE) criterion. MSE minimizes the expected square of the error signal ($e^2$). This mathematical choice is highly effective under a Gaussian noise assumption, where errors are symmetrically distributed and large deviations are exponentially rare.
However, as Marcel Nassar et al. observe in A Factor Graph Approach to Joint OFDM Channel Estimation and Decoding in Impulsive Noise Environments, real-world environments are frequently disrupted by non-Gaussian, heavy-tailed interference. Whether caused by atmospheric discharge, electrical transients in power lines, or marine anomalies in underwater acoustics (Wentao Ma et al. in Sparsity Aware Normalized Least Mean p-power Algorithms with Correntropy Induced Metric Penalty), these disturbances present as "impulsive noise"—rare but massive spikes in signal amplitude.
When an adaptive filter optimized for MSE encounters an impulsive spike, the quadratic penalty forces the error term to explode. The gradient update responds violently to this single outlier, causing the filter's weight estimates to fluctuate wildly. As highlighted by Yi Yu et al. in Two Improved Normalized Subband Adaptive Filter Algorithms with Good Robustness Against Impulsive Interferences and Muhammed O. Sayin et al. in Robust set-membership filtering algorithms against impulsive noise, this sensitivity often leads to catastrophic filter divergence or a complete loss of tracking accuracy.
Saturating the Error with Correntropy
To build resilience against outliers, a core strategy is to replace the quadratic loss function with a localized similarity measure that saturates when errors become excessively large. This is the intuition behind the Maximum Correntropy Criterion (MCC).
Correntropy measures the similarity of two random variables through a kernel function, typically a Gaussian kernel:
$$G_{\sigma}(e) = \exp\left(-\frac{e^2}{2\sigma^2}\right)$$
Instead of minimizing $e^2$, MCC maximizes this kernel value. To understand why this works, consider the behavior of the Gaussian kernel across different error magnitudes. When the error $e$ is small relative to the kernel bandwidth $\sigma$, the Taylor expansion of the exponential function yields $1 - \frac{e^2}{2\sigma^2}$. In this region, maximizing correntropy behaves exactly like minimizing MSE.
However, when a massive impulsive spike occurs ($e \to \infty$), the exponential term decays rapidly toward zero. Crucially, the gradient of this loss function with respect to the error also approaches zero. In effect, the filter treats large outliers as completely uninformative anomalies and "shuts off" its learning mechanism during the spike, preserving its accumulated weights.
Badong Chen et al. demonstrate this stabilization in Constrained Maximum Correntropy Adaptive Filtering, showing that MCC ignores outliers while strictly adhering to linear system constraints. To handle diverse non-Gaussian noise environments, the authors further introduced Generalized Correntropy for Robust Adaptive Filtering, replacing the standard Gaussian kernel with a Generalized Gaussian Density (GGD) kernel to flexibly adapt to different tail behaviors. To overcome the slower convergence speeds typical of localized criteria, Pengcheng Yue et al. proposed a Newtonian-type optimization framework in Newtonian-Type Adaptive Filtering Based on the Maximum Correntropy Criterion, accelerating convergence without sacrificing robustness.
Bounding Gradients and Step Sizes
Beyond kernel-based saturation, another powerful approach to mitigating impulsive noise involves directly bounding the magnitude of the gradient updates or dynamically scaling the filter's step size.
One classic method is to use lower-order norms, such as the $\ell_1$-norm, leading to algorithms like the Sign Least Mean Square (SLMS) or Least Mean Absolute Deviation (LMAD). Because the derivative of the absolute error $|e|$ is simply its sign ($\pm 1$), the gradient step size remains constant regardless of how massive the noise spike is. Lu Lu et al. leverage this property in A Novel Normalized Sign Algorithm for System Identification Under Impulsive Noise Interference, introducing a normalized framework that ensures fast convergence under normal conditions while remaining completely stable during sudden interference bursts.
Alternatively, filters can isolate and suppress noise in the frequency domain. As shown by Yi Yu et al. in Two Improved Normalized Subband Adaptive Filter Algorithms with Good Robustness Against Impulsive Interferences, decomposing signals into individual frequency subbands allows the filter to apply localized scaling functions. When an impulsive spike hits a specific subband, the algorithm instantly shrinks the learning step size for that subband alone, preventing the outlier from corrupting the global filter weights.
Robustness in Sparse and Complex Systems
In many practical applications, such as wireless channel estimation, the underlying system is sparse—meaning only a few coefficients are non-zero. Estimating these systems requires combining robust error criteria with sparsity-inducing penalties (like the $\ell_1$-norm or the Correntropy Induced Metric penalty) to simultaneously reject outliers and zero out irrelevant parameters.
For instance, Guan Gui et al. in Robust Adaptive Sparse Channel Estimation in the Presence of Impulsive Noises integrate zero-attracting and reweighted $\ell_1$-norms into the SLMS framework to achieve accurate channel estimation under non-Gaussian noise. Similarly, Wentao Ma et al. combine sparsity penalties with robust objectives across multiple paradigms: pairing the Correntropy Induced Metric (CIM) with the Least Mean p-power criterion in Sparsity Aware Normalized Least Mean p-power Algorithms with Correntropy Induced Metric Penalty, and with MCC in Maximum correntropy criterion based sparse adaptive filtering algorithms for robust channel estimation under non-Gaussian environments. To maintain tracking performance in recursive frameworks, Yi Yu et al. developed a robust Recursive Least Squares (RLS) variant in Robust Sparsity-Aware RLS Algorithms With Jointly-Optimized Parameters Against Impulsive Noise that jointly optimizes the forgetting factor and sparsity parameters to handle sudden system changes amidst heavy-tailed noise.
This robust philosophy extends to more complex topologies and multidimensional data:
- Noisy Inputs: Traditional filters assume only the output is noisy. When inputs are also corrupted, Wentao Ma et al. developed bias-compensated frameworks in Bias-Compensated Normalized Maximum Correntropy Criterion Algorithm for System Identification with Noisy Input and Sparse Normalized Least Mean Absolute Deviation Algorithm Based on Unbiasedness Criterion for System Identification With Noisy Input, ensuring that input-noise corrections do not collapse when hit by impulsive output noise.
- Distributed Networks: In sensor networks where nodes collaboratively estimate parameters, a single node experiencing impulsive noise can corrupt the entire network. Qing Shi et al. address this in A Robust Diffusion Minimum Kernel Risk-Sensitive Loss Algorithm over Multitask Sensor Networks by utilizing a Minimum Kernel Risk-Sensitive Loss (MKRSL) to isolate local outliers.
- Spatial Context in Images: In image processing, Mingyuan Zhou et al. demonstrate in Dependent Hierarchical Beta Process for Image Interpolation and Denoising that non-parametric Bayesian models can leverage spatial neighborhoods to recognize that isolated, high-amplitude pixel errors do not align with surrounding features, allowing the model to seamlessly interpolate over "spiky" random noise.
Key Foundational Work
Dependent Hierarchical Beta Process for Image Interpolation and Denoising (2011) — contributed 1 concept elements to this direction.
An effective microphone array post-filter in arbitrary environments (2008) — contributed 1 concept elements to this direction.