Abstract
Linear and Quadratic Discriminant Analysis are well-known classical methods but can heavily suffer from non-Gaussian distributions and/or contaminated datasets, mainly because of the underlying Gaussian assumption that is not robust. To fill this gap, this paper presents a new robust discriminant analysis where each data point is drawn by its own arbitrary Elliptically Symmetrical (ES) distribution and its own arbitrary scale parameter. Such a model allows for possibly very heterogeneous, independent but non-identically distributed samples. After deriving a new decision rule, it is shown that maximum-likelihood parameter estimation and classification are very simple, fast and robust compared to state-of-the-art methods.
Introduction
Classical classification methods like Linear and Quadratic Discriminant Analysis (LDA/QDA) are foundational tools in machine learning due to their simplicity and mathematical elegance. However, their reliance on the assumption that data follows a neat Gaussian distribution makes them highly sensitive to outliers and heavy-tailed distributions. In real-world scenarios, data is often "messy" and heterogeneous, violating these rigid assumptions.
To address this, Houdouin et al. introduce FEMDA (Flexible EM-inspired Discriminant Analysis). This framework is designed specifically for datasets where observations are independent but not identically distributed (i.n.i.d.), allowing for extreme flexibility in modeling cluster components by focusing on shared "shape" rather than identical distributions.
The Challenge of Heterogeneous Data
Standard QDA assumes that all points within a class are drawn from the same Gaussian distribution. While robust variants like t-QDA exist, they typically assume a shared distribution (e.g., Student’s t) for all points in a cluster. The authors identify a critical gap: how to handle data where each individual point might follow its own unique distribution and scale. If a dataset contains outliers or points from different sub-families of distributions, these classical models often break down.
The FEMDA Framework
FEMDA introduces a per-point flexibility model. Instead of forcing a single distribution on a cluster, it assumes each observation $x_i$ is drawn from its own Elliptically Symmetrical (ES) distribution with an arbitrary scale parameter $\tau_{i,k}$.
A New Decision Rule
By treating the scale of each point as a "nuisance parameter," the researchers derived a decision rule based on Maximum Likelihood Estimation where the specific generator function of the distribution cancels out. The resulting rule compares the logarithm of squared Mahalanobis distances rather than the distances themselves. This logarithmic transformation is key to its robustness, as it naturally dampens the influence of extreme outliers compared to the linear distances used in classical QDA.
Robust Parameter Estimation
To train the model, FEMDA utilizes iterative fixed-point equations to estimate cluster means and "scatter" (shape) matrices. These estimators automatically assign weights to data points: the further a point is from the cluster center (higher Mahalanobis distance), the less it influences the model. This ensures the cluster parameters remain accurate even in contaminated datasets.
Experimental Evaluation
The performance of FEMDA was validated using synthetic simulations and real-world benchmarks from the UCI repository:
- Synthetic Data: Mixtures of heavy-tailed distributions (Generalized Gaussian, t, and k-distributions) to test resilience to "distributional chaos."
- Spambase: Email classification under varying contamination levels.
- Ecoli & Statlog Landsat: Protein localization and soil type classification based on multi-spectral satellite data.
Results and Discussion
FEMDA demonstrates significant advantages in both accuracy and computational efficiency across various testing scenarios.
Robustness and Accuracy
In synthetic tests with 25% contamination, FEMDA maintained high accuracy while t-QDA's performance degraded. On the Spambase dataset, FEMDA remained stable even at 60% contamination, whereas t-QDA began to fail at 20%. Crucially, FEMDA performs as well as state-of-the-art methods on "clean" data, proving that users do not have to trade off performance in ideal conditions to gain robustness against noise.
Computational Efficiency
A major benefit of FEMDA is its speed. Unlike t-QDA, which requires optimizing degrees-of-freedom parameters, FEMDA’s flexible nature simplifies the parameter space. As shown in the timing benchmarks, FEMDA is significantly faster than complex robust alternatives like RGQDA and offers a more efficient training process by avoiding complex optimization of distribution shape parameters.
Conclusion
FEMDA provides a powerful and efficient framework for classification in heterogeneous environments. By allowing for per-point scale flexibility and utilizing a log-based decision rule, it achieves state-of-the-art robustness without sacrificing speed. While the current model assumes clusters share an elliptical geometry, its success in handling "messy" data suggests broad potential for future extensions in unsupervised learning and other complex data analysis tasks.
Summary updated