When training a machine learning model or designing a signal processing filter, our default choice is almost always Mean Squared Error (MSE) or least squares. This choice is deeply rooted in the assumption of Gaussian noise: under a Gaussian distribution, minimizing the variance of the error is mathematically equivalent to maximizing the likelihood of the data.
But real-world data is rarely perfectly Gaussian. What happens when a system encounters heavy-tailed noise, sudden impulsive bursts, or multimodal distributions?
In these scenarios, MSE fails dramatically. Because it squares the error, a single massive outlier can dominate the loss function and completely skew the optimization process. To build truly resilient systems, researchers use an information-theoretic alternative: the Minimum Error Entropy (MEE) criterion.
The Intuition: Squeezing the Error Distribution
Instead of minimizing the average squared distance of the errors, MEE minimizes the entropy—the uncertainty or dispersion—of the error distribution.
To understand why this works, think about the geometry of the error distribution. MSE tries to pull every single error point closer to zero, penalizing large distances aggressively. MEE, on the other hand, does not care about the absolute magnitude of individual errors as long as the overall error distribution is highly concentrated and predictable. As noted by Seungju Han 0001 et al. in A minimum-error entropy criterion with self-adjusting step-size (MEE-SAS), minimizing error entropy focuses on minimizing the information content of the error signal itself.
MSE Objective: Minimize Variance (Spread around zero)
MEE Objective: Minimize Entropy (Squeeze the distribution into a sharp peak)
By minimizing entropy, we force the probability density function (PDF) of the error to form a sharp, narrow peak. As Badong Chen et al. explain in A Note on the W-S Lower Bound of the MEE Estimation, this approach naturally captures higher-order statistics of the error distribution rather than just its second-order moment (variance). This property provides tighter theoretical bounds on classification and regression errors in non-Gaussian environments, a benefit highlighted by Tülay Adalı et al. in Guest Editorial for Special Issue on Machine Learning for Signal Processing.
Why MEE Outperforms MSE in Complex Noise
The practical superiority of MEE becomes clear when dealing with two specific types of challenging noise: multimodal distributions and structured anomalies.
Multimodal and Heavy-Tailed Noise
In applications like target tracking or state estimation, sensors often experience impulsive or heavy-tailed interference. Traditional Kalman filters rely on MSE and degrade rapidly under these conditions. By integrating the MEE criterion into filtering frameworks—such as the Cubature Kalman Filter proposed by Hanxing Shao et al. in A Hierarchical Adaptive Moment Matching Multiple Model Tracking Method for Hypersonic Glide Target Under Measurement Uncertainty or the smoother developed by Jiacheng He et al. in Minimum Error Entropy Rauch-Tung-Striebel Smoother—the system can effectively suppress complex, multimodal noise distributions where other information-theoretic criteria like Maximum Correntropy falter.
Structured Noise
In computer vision tasks like subspace clustering, noise isn't always random; it can manifest as structured patterns like shadows or occlusions. Yuanman Li et al. demonstrated in Robust Subspace Clustering With Independent and Piecewise Identically Distributed Noise Modeling. that minimizing error entropy allows models to ignore these structured anomalies. Because structured noise possesses lower entropy (more regularity) than purely random fluctuations, an entropy-minimization objective naturally prioritizes the statistical structure over absolute error magnitude, preventing shadows from skewing the clustering boundaries.
Algorithmic Variations and Scaling
While MEE offers powerful robustness, optimizing entropy directly can be computationally expensive because it traditionally requires estimating a continuous PDF from data samples. To bridge this gap, the authors have developed several algorithmic variants:
- Quantization: To maintain computational efficiency in linear regression and causal analysis, Lei Xing et al. introduced a quantized MEE (QMEE) framework in Quantized Minimum Error Entropy Criterion that discretizes the error space, preserving higher-order statistical benefits without heavy computational overhead.
- Sparsity Constraints: In sparse system identification, where many parameters are zero, the MEE criterion has been combined with sparsity-inducing penalties (like the $L_1$-norm or Correntropy Induced Metric). Algorithms developed by Zongze Wu et al. in Proportionate Minimum Error Entropy Algorithm for Sparse System Identification and Minimum Error Entropy Algorithms with Sparsity Penalty Constraints allow adaptive filters to isolate sparse signals even when corrupted by heavy impulsive noise.
- Distributed Learning: For large-scale datasets, Xin Guo et al. proposed a distributed MEE (DMEE) algorithm in Distributed Minimum Error Entropy Algorithms that splits the computation across multiple nodes without requiring inter-node communication, achieving optimal learning rates even in semi-supervised settings.
Modern Frontiers: Deep Networks and Feature Independence
The principles of MEE are also expanding into modern deep learning architectures. For instance, in neuroimaging and brain disorder diagnosis, multi-site data often suffers from high variability and severe outliers. Geng Zhang et al. utilized error entropy minimization in Local-Global Coupling Spiking Graph Transformer for Brain Disorders Diagnosis from Two Perspectives to establish stable decision boundaries across highly variable datasets. Similarly, in neuromorphic computing, Shuangming Yang et al. introduced a Restricted MEE (RMEE) criterion for Spiking Neural Networks in Robust Spike-Based Continual Meta-Learning Improved by Restricted Minimum Error Entropy Criterion, constraining errors to predefined locations to ensure compact and predictable error distributions during continual meta-learning.
An elegant alternative perspective on MEE frames the objective as a problem of statistical dependence. As explored by Shujian Yu et al. in Measuring Dependence with Matrix-based Entropy Functional, a model achieves maximum robustness when the statistical dependence between its input features and its prediction errors is minimized. By ensuring that the residuals retain absolutely no informative patterns from the input data, this approach provides a powerful framework for building models resilient to both non-Gaussian noise and severe covariate shifts.
Theoretical Guarantees and the Bandwidth Tradeoff
The core mechanism behind empirical MEE algorithms typically relies on kernel density estimation (KDE) to approximate the error entropy. This introduces a critical hyperparameter: the kernel bandwidth.
The choice of bandwidth reveals a fascinating theoretical tradeoff. As analyzed by Jun Fan et al. in Consistency Analysis of an Empirical Minimum Error Entropy Algorithm, while a small bandwidth makes minimizing error entropy straightforward, recovering the true underlying regression function in complex, heteroskedastic noise environments requires the bandwidth to scale towards infinity. Unlike variance-minimization approaches, these methods do not require the noise to have finite variance or exponential decay to remain effective, providing a mathematically rigorous foundation for robust information-theoretic learning.
Key Foundational Work
A minimum-error entropy criterion with self-adjusting step-size (MEE-SAS) (2007) — contributed 2 concept elements to this direction.
Guest Editorial for Special Issue on Machine Learning for Signal Processing (2004) — contributed 1 concept elements to this direction.