Gaussian Mixture Models (GMMs) are a cornerstone of semi-parametric probability density function (PDF) estimation, offering a balance between the simplicity of single distributions and the flexibility of non-parametric methods. However, practitioners frequently encounter numerical failures when using the Expectation-Maximization (EM) algorithm to find the Maximum Likelihood Estimate (MLE). This research investigates the root causes of these "crashes," identifying outliers and repeated data points as primary triggers for the "collapsing component" phenomenon.
Methodology
The authors conduct a mathematical analysis of the EM update equations, specifically focusing on the M-step update for kernel width ($\sigma_j$). By decomposing the variance update equation, they isolate the influence of individual data points. This theoretical framework is validated using an artificial 2D dataset consisting of 300 samples from two uniform distributions. Two specific scenarios are tested:
- Outlier Scenario: A single point placed far from the main clusters.
- Repetition Scenario: A single data point repeated $K$ times (5% of the total dataset).
A 5-component GMM is then fitted to these datasets to observe the algorithm's behavior across iterations.
The Mechanics of Component Collapse
The study identifies "isolation" as the fundamental cause of EM failure. Isolation can be physical, as in the case of outliers, or relative, as seen with repeated data points.
- The Snowball Effect: As a component's width ($\sigma$) decreases, its statistical "reach" shrinks exponentially. This causes the component to ignore neighboring data, leading to a further reduction in variance. This positive feedback loop eventually drives $\sigma$ toward zero.
- Outliers vs. Repetitions: Outliers act as "magnets," pulling a component's center away from the main distribution until it collapses onto the isolated point. Repeated data is more subtle; the high density of identical points causes a component to "specialize" in them, eventually ignoring the rest of the data even if it is nearby.
Experimental Results
The experimental results demonstrate that while outliers lead to a "poor" local maximum where a component is wasted, repeated samples cause the algorithm to effectively crash as the log-likelihood spikes toward infinity.
As shown in Figure 1, Gaussian ellipses can be seen shrinking into singular points over 20–30 iterations. This physical collapse corresponds to specific signatures in the log-likelihood function.
In the case of repeated samples (Figure 6b), the likelihood does not plateau but instead rises sharply as the component collapses, indicating a failure to represent the true underlying PDF.
Conclusion and Limitations
While this research provides a rigorous diagnosis of EM failure modes, it highlights several areas for future work. The study is limited to 2D space; in higher dimensions, data sparsity likely exacerbates these isolation problems. Furthermore, the paper focuses on identifying the problem rather than proposing mitigation strategies, such as regularization or Bayesian priors, which remain essential for robust GMM implementation in practice.
Summary updated