Combining mixture models with linear mixing updates: multilayer image segmentation and synthesis

Jonathan Vacher, Ruben Coen Cagli

Abstract

Probabilistic finite mixture models are widely used for unsupervised clustering. These models can often be improved by adapting them to the topology of the data. For instance, in order to classify spatially adjacent data points similarly, it is common to introduce a Laplacian constraint on the posterior probability that each data point belongs to a class. Alternatively, the mixing probabilities can be treated as free parameters, while assuming Gauss-Markov or more complex priors to regularize those mixing probabilities. However, these approaches are constrained by the shape of the prior and often lead to complicated or intractable inference. Here, we propose a new parametrization of the Dirichlet distribution to flexibly regularize the mixing probabilities of over-parametrized mixture distributions. Using the Expectation-Maximization algorithm, we show that our approach allows us to define any linear update rule for the mixing probabilities, including spatial smoothing regularization as a special case. We then show that this flexible design can be extended to share class information between multiple mixture models. We apply our algorithm to artificial and natural image segmentation tasks, and we provide quantitative and qualitative comparison of the performance of Gaussian and Student-t mixtures on the Berkeley Segmentation Dataset. We also demonstrate how to propagate class information across the layers of deep convolutional neural networks in a probabilistically optimal way, suggesting a new interpretation for feedback signals in biological visual systems. Our flexible approach can be easily generalized to adapt probabilistic mixture models to arbitrary data topologies.

Introduction

Standard probabilistic mixture models, such as Gaussian Mixture Models (GMMs), typically operate under the assumption that data points are independent and identically distributed (i.i.d.). In domains like image segmentation, this assumption is fundamentally flawed because the spatial topology of the data is highly informative—neighboring pixels are significantly more likely to belong to the same semantic class. While existing methods have attempted to incorporate spatial constraints via Markov Random Fields or Dirichlet Processes, they often result in non-linear, computationally expensive, or mathematically intractable inference. This paper introduces FlexMM (Flexibly regularized Mixture Models), a framework that integrates data topology into unsupervised clustering through a novel parametrization of the Dirichlet distribution.

Methodology

The core innovation of FlexMM lies in its treatment of mixing probabilities—the prior probability that a data point belongs to a specific class. Instead of treating these as fixed parameters, the authors model them as random variables following a Dirichlet distribution.

Crucially, they parametrize this distribution such that its parameters depend linearly on the class assignments of other data points. This design choice ensures that the M-step of the Expectation-Maximization (EM) algorithm remains a simple linear update. For image segmentation, this translates to a prior for a pixel that is a weighted average of the posterior probabilities of its neighbors. To handle the loops introduced in the graphical representation (where classes depend on priors that in turn depend on other classes), the authors employ factor graph theory to guarantee that the EM algorithm functions as a valid ascent algorithm, consistently increasing the log-posterior.

Comparison of graphical models: (a) Standard mixture model, (b) Location Dependent Dirichlet Process, (c) Spatially varying mixture models, and (d) the proposed FlexMM mixture model.

Hierarchical Integration and CNNs

The framework is further extended to a hierarchical setting, allowing for the sharing of class information across multiple mixture models. This is particularly useful for integrating features from different layers of a deep convolutional neural network (CNN), such as VGG19. By propagating class information across layers, the model can combine low-level texture information with high-level semantic features. The integration is uncertainty-weighted: layers with high local variance or low spatial resolution contribute less to the final segmentation, ensuring a probabilistically optimal fusion of information.

Experimental Results

The authors evaluated FlexMM on synthetic 2D grids and the Berkeley Segmentation Dataset (BSD500). Key findings include:

  • Robustness of Student-t Models: For natural images, Student-t Mixture Models (SMM) significantly outperformed GMMs. The heavy tails of the Student-t distribution provide robustness against outliers and complex noise inherent in visual data.
  • Performance Gains: FlexMM outperformed classical unsupervised algorithms like K-Means, Birch, and MeanShift on BSD500 in both region-based (aRI) and boundary-based (F-score) metrics.
  • Semantic Grouping: Integrating deep features from VGG19 improved the Adjusted Rand Index (aRI), as higher-level layers helped group large, semantically related regions that low-level RGB features would otherwise fragment.
  • Computational Efficiency: The linear update rules prevented the numerical instabilities and high costs associated with non-linear state-of-the-art methods, allowing for rapid convergence.

Conclusion and Future Work

FlexMM provides a flexible, efficient way to adapt mixture models to arbitrary data topologies. While it represents a significant step forward for unsupervised segmentation, it still performs below human-level benchmarks and supervised deep learning models. Future research directions include automating the selection of the number of mixture components ($K$), refining the integration of high-dimensional deep features without lossy dimensionality reduction, and incorporating explicit object class labels to further bridge the gap between low-level grouping and high-level recognition.

Summary updated

Abstract

Probabilistic finite mixture models are widely used for unsupervised clustering. These models can often be improved by adapting them to the topology of the data. For instance, in order to classify spatially adjacent data points similarly, it is common to introduce a Laplacian constraint on the posterior probability that each data point belongs to a class. Alternatively, the mixing probabilities can be treated as free parameters, while assuming Gauss-Markov or more complex priors to regularize those mixing probabilities. However, these approaches are constrained by the shape of the prior and often lead to complicated or intractable inference. Here, we propose a new parametrization of the Dirichlet distribution to flexibly regularize the mixing probabilities of over-parametrized mixture distributions. Using the Expectation-Maximization algorithm, we show that our approach allows us to define any linear update rule for the mixing probabilities, including spatial smoothing regularization as a special case. We then show that this flexible design can be extended to share class information between multiple mixture models. We apply our algorithm to artificial and natural image segmentation tasks, and we provide quantitative and qualitative comparison of the performance of Gaussian and Student-t mixtures on the Berkeley Segmentation Dataset. We also demonstrate how to propagate class information across the layers of deep convolutional neural networks in a probabilistically optimal way, suggesting a new interpretation for feedback signals in biological visual systems. Our flexible approach can be easily generalized to adapt probabilistic mixture models to arbitrary data topologies.
BibTeX
@article{vacher2022combining,
  author = {Jonathan Vacher and Ruben Coen Cagli},
  title = {{Combining mixture models with linear mixing updates: multilayer image segmentation and synthesis}},
  journal = {Neural Networks},
  year = {2022},
  doi = {10.1016/j.neunet.2022.02.010},
}
Figure