Maximum Correntropy Criterion with Variable Center

Badong Chen, Xin Wang, Yingsong Li, José C. Príncipe

Abstract

Correntropy is a local similarity measure defined in kernel space and the maximum correntropy criterion (MCC) has been successfully applied in many areas of signal processing and machine learning in recent years. The kernel function in correntropy is usually restricted to the Gaussian function with center located at zero. However, zero-mean Gaussian function may not be a good choice for many practical applications. In this study, we propose an extended version of correntropy, whose center can locate at any position. Accordingly, we propose a new optimization criterion called maximum correntropy criterion with variable center (MCC-VC). We also propose an efficient approach to optimize the kernel width and center location in MCC-VC. Simulation results of regression with linear in parameters (LIP) models confirm the desirable performance of the new method.

Introduction

In supervised learning, minimizing the Mean Squared Error (MSE) is a standard approach. While MSE is optimal when noise follows a Gaussian distribution, it is highly sensitive to outliers and heavy-tailed noise. To address this, the Maximum Correntropy Criterion (MCC) has emerged as a robust alternative. MCC uses a Gaussian kernel to measure similarity between predicted and target values, effectively down-weighting large outliers.

However, standard MCC has a significant limitation: the Gaussian kernel is typically fixed with a center at zero. If the noise in a system is biased (non-zero mean), a zero-centered kernel cannot accurately capture the error distribution, leading to sub-optimal performance. This paper proposes Maximum Correntropy Criterion with Variable Center (MCC-VC), allowing the kernel to adapt to shifted noise distributions.

Methods

The proposed MCC-VC objective function is defined as:

$$V_{\sigma,c}(T, Y) = E[G_\sigma(e - c)]$$

where $e$ is the error, $\sigma$ is the kernel width, and $c$ is the variable center.

Key Technical Contributions

  • PDF Matching: Maximizing MCC-VC is mathematically equivalent to aligning the Gaussian kernel with the error's Probability Density Function (PDF).
  • Optimization Strategy: The authors developed an efficient approach to jointly optimize model parameters, kernel width ($\sigma$), and center location ($c$), avoiding manual trial-and-error.
  • Fixed-Point Iteration: For "Linear In Parameters" (LIP) models, a fixed-point iteration algorithm was derived. Unlike MSE which only considers the second moment, MCC-VC involves all even moments of the error about the center $c$, capturing significantly more information about the noise structure.

Results and Discussion

The researchers evaluated MCC-VC using synthetic linear regression with various noise types and real-world benchmarks from the UCI repository.

Synthetic Noise Performance

In tests with various noise types (Gaussian, Laplace, Chi-square), MCC-VC consistently outperformed standard methods. Notably, in non-zero-mean Gaussian noise, MCC-VC achieved an RMSE of 0.0505, vastly outperforming standard MCC (0.1375) and MMSE (1.2214). This demonstrates the critical advantage of a movable kernel center in biased environments.

Benchmark Performance

When applied to Extreme Learning Machines (ELM), MCC-VC consistently yielded lower testing RMSEs compared to regularized ELM (RELM) and standard robust ELM (ELM-RCC) across multiple datasets.

RMSE performance comparison across real-world datasets

Visual Confirmation of PDF Matching

The effectiveness of MCC-VC is visually evident in how the Gaussian kernel "tracks" the error distribution. By allowing the center to vary, the algorithm aligns its peak with the actual peak of the noise PDF, effectively acting as a local PDF estimator.

Error distribution and Gaussian kernel function matching

Conclusion

MCC-VC addresses a fundamental limitation of standard MCC by introducing a variable kernel center. This adaptation allows for significantly better performance in non-zero-mean noise environments. While currently optimized for LIP models, future research could extend this concept to deep learning architectures and other kernel-based similarity measures.

Summary updated

Abstract

Correntropy is a local similarity measure defined in kernel space and the maximum correntropy criterion (MCC) has been successfully applied in many areas of signal processing and machine learning in recent years. The kernel function in correntropy is usually restricted to the Gaussian function with center located at zero. However, zero-mean Gaussian function may not be a good choice for many practical applications. In this study, we propose an extended version of correntropy, whose center can locate at any position. Accordingly, we propose a new optimization criterion called maximum correntropy criterion with variable center (MCC-VC). We also propose an efficient approach to optimize the kernel width and center location in MCC-VC. Simulation results of regression with linear in parameters (LIP) models confirm the desirable performance of the new method.
BibTeX
@article{chen2019maximum,
  author = {Badong Chen and Xin Wang and Yingsong Li and José C. Príncipe},
  title = {{Maximum Correntropy Criterion with Variable Center}},
  journal = {IEEE Signal Processing Letters},
  year = {2019},
  url = {https://openreview.net/forum?id=nAIbU1ktkl},
}
Figure