Maximum Correntropy Criterion With Variable Center

Badong Chen, Xin Wang, Yingsong Li, José C. Príncipe

TL;DR

Standard Maximum Correntropy Criterion (MCC) is a powerful tool for robust learning, but it typically assumes that error distributions are centered at zero. This paper introduces MCC with a Variable Center (MCC-VC), which allows the kernel to adapt to shifted or biased noise distributions, significantly improving performance and training efficiency in non-Gaussian environments.

What Problem Does This Paper Address?

In supervised learning, we usually try to minimize the Mean Squared Error (MSE). While MSE works perfectly when noise follows a normal (Gaussian) distribution, it is notoriously sensitive to outliers and heavy-tailed noise. To solve this, researchers often use the Maximum Correntropy Criterion (MCC), which uses a Gaussian kernel to measure similarity between predicted and target values, effectively "ignoring" large outliers.

However, standard MCC has a major limitation: the Gaussian kernel is fixed at a center of zero. If the noise in a system has a non-zero mean (i.e., it is biased), a zero-centered kernel cannot accurately match the error distribution. This leads to sub-optimal learning. The authors tackle this by proposing a version of correntropy where the center of the kernel can move to any position, allowing the model to "find" the true center of the noise.

Methods

The authors define a new objective function, $V_{\sigma,c}(T, Y) = E[G_\sigma(e - c)]$, where $e$ is the error, $\sigma$ is the kernel width, and $c$ is the variable center.

Key technical contributions include:

  • PDF Matching: They demonstrate that maximizing this new correntropy is mathematically equivalent to making the Gaussian kernel match the error's Probability Density Function (PDF) as closely as possible.
  • Optimization Strategy: Instead of manual trial-and-error, they propose an efficient approach to jointly optimize the model parameters, the kernel width ($\sigma$), and the center location ($c$).
  • Fixed-Point Iteration: For "Linear In Parameters" (LIP) models, they derive a fixed-point iteration algorithm. This is generally more stable and faster than standard gradient descent for this specific type of cost function.
  • LIP-MCC-VC Algorithm: A structured algorithm that computes errors, updates the kernel parameters via PDF matching, and then updates the model weights iteratively.

Datasets

The researchers evaluated their method using two main approaches:

  1. Synthetic Linear Regression: A 2D linear system where they injected four types of noise: zero-mean Gaussian, non-zero-mean Gaussian, Laplace, and Chi-square.
  2. UCI Machine Learning Repository Benchmarks: They tested the algorithm on seven real-world datasets: Servo, Airfoil, Concrete, Housing, Yacht, Wine-red, and Slump. These were used in conjunction with Extreme Learning Machines (ELM), a type of neural network where only the output weights are trained.

Key Insights and Ideas

  • The "Aha" Moment on PDF Matching: The paper shifts the view of MCC from just a "robust loss function" to a "local PDF estimator." By allowing the center to vary, the algorithm essentially aligns its "window of reliability" with the actual peak of the error distribution.
  • Handling Bias: Standard MCC fails when noise is biased (non-zero mean). MCC-VC treats the bias as a parameter to be learned rather than a problem to be ignored.
  • Moment Analysis: The authors show that MCC-VC involves all even moments of the error about the center $c$. This explains why it captures much more information about the noise structure than MSE (which only looks at the second moment).
  • Efficiency over Manual Tuning: By solving an optimization problem to find $\sigma$ and $c$, the authors found they could actually save training time compared to standard MCC, which often requires extensive manual "trial-and-error" to find the right hyperparameters.

Key Results

  • Superior Robustness: In the non-zero-mean Gaussian noise test, MCC-VC achieved an RMSE of 0.0505, vastly outperforming standard MCC (0.1375) and MMSE (1.2214).
  • Benchmark Performance: On the Yacht dataset, the MCC-VC based ELM achieved a testing RMSE of 0.0232, compared to 0.0769 for the standard robust ELM (ELM-RCC) and 0.0823 for regularized ELM.
  • Visual Confirmation: The paper includes plots showing the Gaussian kernel "tracking" the error distribution over iterations, confirming that the variable center accurately migrates to the peak of the noise PDF.

Limitations and Future Work

  • Model Scope: The current derivation and fixed-point iteration are specifically tailored for Linear In Parameters (LIP) models. While this includes ELMs and RBF networks, extending this to deep, multi-layer architectures using standard backpropagation would require further derivation.
  • Search Space: The optimization of $\sigma$ and $c$ currently relies on searching through predefined sets ($S$ and $C$). While more efficient than trial-and-error, it still requires the user to define reasonable bounds for these parameters.
  • Future Directions: The authors suggest that this "variable center" concept could be applied to other kernel-based similarity measures beyond the Gaussian kernel.

Summary updated

Abstract

Correntropy is a local similarity measure defined in kernel space, and the maximum correntropy criterion (mcc) has been successfully applied in many areas of signal processing and machine learning in recent years. The kernel function in correntropy is usually restricted to the Gaussian function with the center located at zero. However, the zero-mean Gaussian function may not be a good choice for many practical applications. In this letter, we propose an extended version of correntropy, whose center can be located at any position. Accordingly, we propose a new optimization criterion called maximum correntropy criterion with variable center (MCC-VC). We also propose an efficient approach to optimize the kernel width and center location in the MCC-VC. Simulation results of regression with linear-in-parameter (LIP) models confirm the desirable performance of the new method.
BibTeX
@article{chen2019maximum,
  author = {Badong Chen and Xin Wang and Yingsong Li and José C. Príncipe},
  title = {{Maximum Correntropy Criterion With Variable Center}},
  journal = {IEEE Signal Processing Letters},
  year = {2019},
  url = {https://openreview.net/forum?id=nAIbU1ktkl},
}

Loading related content...

Figure