Flexibly Regularized Mixture Models and Application to Image Segmentation

Jonathan Vacher, Claire Launay, Ruben Coen-Cagli

Abstract

Probabilistic finite mixture models are widely used for unsupervised clustering. These models can often be improved by adapting them to the topology of the data. For instance, in order to classify spatially adjacent data points similarly, it is common to introduce a Laplacian constraint on the posterior probability that each data point belongs to a class. Alternatively, the mixing probabilities can be treated as free parameters, while assuming Gauss-Markov or more complex priors to regularize those mixing probabilities. However, these approaches are constrained by the shape of the prior and often lead to complicated or intractable inference. Here, we propose a new parametrization of the Dirichlet distribution to flexibly regularize the mixing probabilities of over-parametrized mixture distributions. Using the Expectation-Maximization algorithm, we show that our approach allows us to define any linear update rule for the mixing probabilities, including spatial smoothing regularization as a special case. We then show that this flexible design can be extended to share class information between multiple mixture models. We apply our algorithm to artificial and natural image segmentation tasks, and we provide quantitative and qualitative comparison of the performance of Gaussian and Student-t mixtures on the Berkeley Segmentation Dataset. We also demonstrate how to propagate class information across the layers of deep convolutional neural networks in a probabilistically optimal way, suggesting a new interpretation for feedback signals in biological visual systems. Our flexible approach can be easily generalized to adapt probabilistic mixture models to arbitrary data topologies.

Introduction

Clustering is a fundamental problem in machine learning, yet standard approaches often struggle when data possesses a specific underlying topology. For example, in image segmentation, neighboring pixels are highly likely to share the same class label. Conventional mixture models, such as Gaussian Mixture Models (GMMs), typically treat each data point as an independent and identically distributed (i.i.d.) sample, ignoring these critical spatial relationships.

Existing methods to incorporate topology often rely on adding Laplacian constraints or complex priors like Gauss-Markov random fields. While effective in some contexts, these methods frequently result in "loopy" graphical models where inference becomes mathematically intractable or requires computationally expensive, non-linear optimization. In their paper, "Flexibly Regularized Mixture Models and Application to Image Segmentation," Jonathan Vacher and colleagues introduce FlexMM, a framework that provides a mathematically elegant and computationally efficient way to integrate data topology into mixture models.

The FlexMM Framework

The core innovation of FlexMM is a novel re-parametrization of the Dirichlet distribution. By defining the parameters of the Dirichlet prior as a linear function of the class assignments of other samples, the authors ensure that the M-step in the Expectation-Maximization (EM) algorithm remains a simple linear update rule. This linearity is a significant advantage, providing numerical stability and computational speed compared to previous spatially-varying models that often required solving complex non-linear equations.

Comparison of graphical models: (a) Standard mixture model, (b) Location Dependent Dirichlet Process, (c) Spatially varying mixture models, and (d) the proposed FlexMM mixture model.

To maintain mathematical rigor despite the introduction of dependencies between samples (which creates loops in the graphical model), the researchers utilized factor graph theory. They proved that their modified EM algorithm remains an "ascent algorithm," guaranteeing that the model's likelihood improves at every iteration. Furthermore, the framework is extensible; it can handle multiple "layers" of data, allowing information to flow both horizontally (between spatial neighbors) and vertically (between different levels of abstraction, such as layers in a deep neural network).

Experimental Validation

The authors evaluated FlexMM across three distinct scenarios:

  1. Synthetic Data: Using 2D grids of Gaussian samples, they demonstrated that FlexMM could recover ground-truth segments in high-noise environments where standard GMMs failed.
  2. Berkeley Segmentation Dataset (BSD500): On this benchmark for natural image segmentation, FlexMM with spatial smoothing consistently outperformed classical unsupervised methods like KMeans, Birch, and MeanShift in terms of Adjusted Rand Index (aRI) and F-scores.
  3. Hierarchical Features (VGG-19): By applying FlexMM to features from various layers of a pre-trained VGG-19 network, the authors showed that the hierarchical version of the model (Model b in Figure 5) could effectively use high-level "object" information from deep layers to refine low-level "texture" noise in shallower layers.

Key Insights

A major finding was that Student-t Mixture Models (SMM) significantly outperformed Gaussian ones for natural images. The heavy tails of the Student-t distribution are better suited to handling the outliers and "messy" data inherent in natural scenes.

Additionally, the hierarchical model introduces a concept of uncertainty-weighted feedback. Information propagated between layers is weighted by the model's confidence. If a deep layer is certain about a segment while a shallow layer is confused by local texture, the deep layer's signal dominates. This provides a compelling probabilistic interpretation of the feedback signals observed in biological visual systems.

Conclusion and Future Work

FlexMM offers a versatile toolbox for regularizing mixture models on structured data. While it represents a significant step forward in unsupervised clustering, it does not yet match the performance of human observers or state-of-the-art supervised deep learning models in image segmentation. Future work may involve incorporating explicit object classes to bias priors, moving toward semi-supervised approaches, and further refining the selection of hyperparameters such as the number of components and regularization kernels. Nevertheless, the framework's combination of mathematical rigor and practical flexibility makes it a powerful tool for adapting probabilistic models to arbitrary data topologies.

Summary updated

Abstract

Probabilistic finite mixture models are widely used for unsupervised clustering. These models can often be improved by adapting them to the topology of the data. For instance, in order to classify spatially adjacent data points similarly, it is common to introduce a Laplacian constraint on the posterior probability that each data point belongs to a class. Alternatively, the mixing probabilities can be treated as free parameters, while assuming Gauss-Markov or more complex priors to regularize those mixing probabilities. However, these approaches are constrained by the shape of the prior and often lead to complicated or intractable inference. Here, we propose a new parametrization of the Dirichlet distribution to flexibly regularize the mixing probabilities of over-parametrized mixture distributions. Using the Expectation-Maximization algorithm, we show that our approach allows us to define any linear update rule for the mixing probabilities, including spatial smoothing regularization as a special case. We then show that this flexible design can be extended to share class information between multiple mixture models. We apply our algorithm to artificial and natural image segmentation tasks, and we provide quantitative and qualitative comparison of the performance of Gaussian and Student-t mixtures on the Berkeley Segmentation Dataset. We also demonstrate how to propagate class information across the layers of deep convolutional neural networks in a probabilistically optimal way, suggesting a new interpretation for feedback signals in biological visual systems. Our flexible approach can be easily generalized to adapt probabilistic mixture models to arbitrary data topologies.
BibTeX
@article{vacher2022flexibly,
  author = {Jonathan Vacher and Claire Launay and Ruben Coen-Cagli},
  title = {{Flexibly Regularized Mixture Models and Application to Image Segmentation}},
  journal = {Neural Networks},
  year = {2022},
  doi = {10.1016/j.neunet.2022.02.010},
}
Figure