Flexible and Robust Bayesian Classification by Finite Mixture Models

Cédric Archambeau, Frédéric Vrins, Michel VerleysenESANN 2004 · OpenReview

Abstract

Finite Mixture Models (FMs) are a cornerstone of statistical pattern recognition, yet they are frequently plagued by numerical instability and sensitivity to outliers. This paper introduces a regularized classification framework designed to mitigate these issues. By incorporating a regularized Mahalanobis distance and leveraging the heavy-tailed properties of Student-$t$ distributions, the proposed approach ensures numerical stability during training and provides robust performance in noisy environments.

Introduction

In Bayesian classification, Finite Mixture Models are used to estimate the underlying probability density of data classes. While flexible, standard Gaussian Mixture Models (FGM) often fail in practical applications due to two primary factors: numerical collapse and outlier sensitivity. Numerical collapse occurs when a mixture component's variance shrinks to zero, causing the likelihood to approach infinity and the Expectation-Maximization (EM) algorithm to fail. Furthermore, the thin tails of the Gaussian distribution make these models highly susceptible to "dirty" data, where even a few outliers can drastically skew class parameters.

Proposed Methodology

To address these limitations, the authors propose two significant enhancements to the standard FM framework:

1. Regularized Mahalanobis Distance

Instead of relying solely on the standard Mahalanobis distance, which can lead to singular covariance matrices, the authors introduce a convex combination of Mahalanobis and Euclidean distances. This regularization is controlled by a tuning parameter $\lambda$:

  • Euclidean Component: Encourages hyperspherical clusters, preventing components from becoming infinitely thin.
  • Stabilization Parameter ($\epsilon$): Ensures that covariance matrices remain non-singular and invertible, effectively preventing the "isolation" problem where a component collapses onto a single data point.

2. Regularized Student-$t$ Mixture Models (RFTM)

While the regularized Gaussian model (RFGM) improves stability, it remains sensitive to outliers. The authors extend the regularization to Student-$t$ distributions. The Student-$t$ distribution possesses "heavy tails," allowing it to accommodate atypical observations without shifting the cluster centers as aggressively as a Gaussian model would. This inherent robustness makes the RFTM a powerful tool for processing corrupted datasets.

Experimental Evaluation

The framework was evaluated using Ripley’s synthetic 2D dataset, corrupted with two types of noise to test robustness: additive Gaussian noise and uniform random outliers (atypical observations).

Ripley's synthetic dataset under different noise conditions: (a) original, (b) additive Gaussian noise, and (c) uniform outliers.

As shown in Figure 1, the noise levels were significant enough to challenge standard classification boundaries. In these scenarios, unregularized models frequently failed to converge due to numerical instability.

Results and Discussion

The performance of the Regularized Gaussian (RFGM) and Regularized Student-$t$ (RFTM) models was compared against the theoretical Bayes optimal error rate of 8%.

Correct classification rate as a function of the regularization parameter λ for RFGM (blue) and RFTM (red) models under different noise conditions.

Key findings include:

  • Superior Robustness: In the presence of Gaussian noise, the RFTM achieved a 9.6% error rate, outperforming the RFGM's 10.8%. Under uniform outliers, both models remained highly effective, with RFTM slightly leading at 9.3%.
  • Optimal Regularization: Figure 5 illustrates that the classification rate is highly dependent on the parameter $\lambda$. The RFTM generally achieved peak performance with lower values of $\lambda$ than the RFGM, indicating that the Student-$t$ distribution's inherent flexibility allows it to capture the true data shape more accurately while remaining stable.
  • Stability as a Prerequisite: The most critical result was that the regularized models remained functional under noise levels that caused standard FGM and FTM models to crash entirely.

Conclusion

The integration of regularized distances with Student-$t$ mixture models provides a robust and numerically stable framework for Bayesian classification. While the current approach relies on exhaustive search for hyperparameter tuning (such as $\lambda$ and the degrees of freedom $\nu$), it sets a strong foundation for future work in automated parameter estimation and scaling to higher-dimensional datasets.

Summary updated

BibTeX
@inproceedings{archambeau2004flexible,
  author = {Cédric Archambeau and Frédéric Vrins and Michel Verleysen},
  title = {{Flexible and Robust Bayesian Classification by Finite Mixture Models}},
  booktitle = {ESANN},
  year = {2004},
  url = {https://openreview.net/forum?id=BMM-czmkk79},
}
Figure