Revisiting the Robustness of the Minimum Error Entropy Criterion: A Transfer Learning Case Study

Luis Pedro Silvestrin, Shujian Yu, Mark Hoogendoorn

TL;DR

Swapping standard Mean Squared Error (MSE) for a matrix-based Minimum Error Entropy (MEE) loss function provides inherent, powerful robustness against both non-Gaussian noise and covariate shift in regression tasks. By simply integrating MEE into classic transfer learning workflows like fine-tuning and linear probing, the authors achieve competitive, state-of-the-art performance on complex time-series datasets without needing complicated architectural modifications.

What Problem Does This Paper Address?

Real-world machine learning models frequently fail when deployed because of distributional shift—a mismatch between the training data distribution and the test data distribution. A common type of shift is covariate shift, where the input feature distribution $p(x)$ changes but the relationship between features and labels $p(y|x)$ stays the same (e.g., sensor drift over time or seasonal weather changes).

To make matters worse, real-world data is rarely clean. Standard regression models rely heavily on Mean Squared Error (MSE) or Mean Absolute Error (MAE) losses, which implicitly assume that data noise follows a clean Gaussian or Laplacian distribution. When faced with real-world, non-Gaussian noise (like heavy-tailed outliers or skewed distributions), these models degrade rapidly. While specialized transfer learning algorithms exist to combat covariate shift, they typically require complex training paradigms (like adversarial training or boosting ensembles) that are notoriously difficult to tune and implement.

The authors bridge this gap by revisiting a classic objective from statistical signal processing—the Minimum Error Entropy (MEE) criterion—and adapting it for deep transfer learning regression.

Methods

The authors introduce a framework that replaces MSE with MEE within standard transfer learning pipelines (specifically fine-tuning and linear probing). Their approach includes several key technical components:

  • Matrix-Based Entropy Formulation: Traditional MEE relies on Kernel Density Estimation (KDE) to approximate error distributions, which scales poorly to high dimensions and deep networks. The authors utilize a differentiable, matrix-based formulation of Rényi’s $\alpha$-order entropy ($\alpha=2$) that completely bypasses density estimation, making it highly compatible with deep neural networks like Temporal Convolutional Networks (TCNs).
  • Theoretical Proof of Covariate Shift Robustness: The authors provide a novel theoretical justification showing that minimizing error entropy mathematically minimizes the mutual information $I(x; e)$ between the inputs ($x$) and the prediction errors ($e$). By forcing the error distribution to be independent of the input distribution, the model naturally becomes resilient to changes in $p(x)$ (covariate shift).
  • Correcting Model Bias: MEE is shift-invariant, meaning that two models whose predictions differ only by a constant value will yield the exact same entropy loss. To prevent the model from converging to a systematically biased solution, the authors introduce a post-optimization bias correction step that shifts predictions by the average residual calculated on the target training data.
  • Automated Kernel Tuning: MEE introduces an extra hyperparameter: the RBF kernel width ($\sigma$). The authors eliminate manual tuning by implementing the "median-rule" heuristic, automatically setting $\sigma$ based on the median pair-wise distance between initial residuals.

Datasets

The authors evaluated their approach using both controlled synthetic environments and real-world benchmarks:

  • Synthetic Linear Regression Data: Used to precisely simulate varying degrees of covariate shift under three distinct noise regimes: shifted exponential noise, mixed Gaussian noise, and Laplace noise.
  • Real-World Time-Series Benchmarks: Five distinct transfer learning regression tasks derived from three popular repositories (summarized in the table below):
  • NASA Turbofan Dataset (NTS, NT1, NT2, NT3): Predicting the remaining useful life of engines operating under different environmental conditions and fault modes.
  • Beijing Air Quality Dataset (PMS, PMT): Predicting PM10 air concentrations across different years.
  • Bike Sharing Dataset (BKS, BKT): Predicting rental demand across distinct seasonal shifts (e.g., training on fall/winter/spring and testing on summer).
Dataset details including sample sizes, window sizes, and feature counts

Key Insights and Ideas

  • The Independence Trick: The core "aha moment" is that decoupling error behavior from input behavior inherently insulates a model from input distribution changes. If the error distribution $p(e)$ is independent of $p(x)$, a shift in $p(x)$ won't alter the error profile.
  • Simplicity Beats Complexity: Instead of building complex adversarial networks or multi-stage boosting ensembles, simply changing the loss function in a standard fine-tuning pipeline yields superior robustness.
  • Eliminating the Hyperparameter Headache: The matrix-based MEE combined with the automated median-rule heuristic removes the traditional barriers to using entropy losses in deep learning. As shown in the error density visualization below, the automated median rule ($\sigma=1$) perfectly concentrates prediction errors around zero compared to poorly tuned alternatives.
Comparison of error densities using MEE with different kernel sizes

Key Results

  • Synthetic Evaluation: Under severe covariate shift and non-Gaussian noise, the MEE loss demonstrated a significantly slower error increase compared to MSE, MAE, and the state-of-the-art Hilbert-Schmidt Independence Criterion (HSIC) loss. This was particularly striking in the presence of shifted exponential noise and mixed Gaussian noise.
Shifted Exponential Noise Mixed Gaussian Noise Laplace Noise
MEE performance under shifted exponential noise MEE performance under mixed Gaussian noise MEE performance under Laplace noise
MEE performance under shifted exponential noise
MEE performance under mixed Gaussian noise
MEE performance under Laplace noise
  • Linear Probing: When freezing the feature extractor and only training the output layer, MEE achieved significantly lower target squared error than MSE, MAE, and HSIC in 4 out of the 5 real-world time-series datasets.
  • Fine-Tuning vs. State-of-the-Art: When compared against specialized, complex transfer learning algorithms tailored for regression, the simple MEE-based fine-tuning approach outperformed the adversarial WANN framework across all 5 real-world tasks and beat the boosting-based TrAdaBoost.R2 (TRB) method in 4 out of 5 tasks.

Limitations and Future Work

While MEE proved highly effective, the authors note that baseline comparison algorithms like WANN and TRB were originally conceived for tabular data rather than sequential time-series data. A promising future research direction includes natively adapting these specialized transfer learning frameworks to utilize MEE instead of MSE as their internal objective function. Additionally, the authors suggest that further theoretical studies are needed to fully map out how MEE impacts general out-of-distribution generalization.

Summary updated

Abstract

Coping with distributional shifts is an important part of transfer learning methods in order to perform well in real-life tasks. However, most of the existing approaches in this area either focus on an ideal scenario in which the data does not contain noises or employ a complicated training paradigm or model design to deal with distributional shifts. In this paper, we revisit the robustness of the minimum error entropy (MEE) criterion, a widely used objective in statistical signal processing to deal with non-Gaussian noises, and investigate its feasibility and usefulness in real-life transfer learning regression tasks, where distributional shifts are common. Specifically, we put forward a new theoretical result showing the robustness of MEE against covariate shift. We also show that by simply replacing the mean squared error (MSE) loss with the MEE on basic transfer learning algorithms such as fine-tuning and linear probing, we can achieve competitive performance with respect to state-of-the-art transfer learning algorithms. We justify our arguments on both synthetic data and 5 real-world time-series data.
BibTeX
@inproceedings{silvestrin2023revisiting,
  author = {Luis Pedro Silvestrin and Shujian Yu and Mark Hoogendoorn},
  title = {{Revisiting the Robustness of the Minimum Error Entropy Criterion: A Transfer Learning Case Study}},
  booktitle = {ECAI},
  year = {2023},
}
Figure