Optimal Transport Mixing of Gaussian Texture Models

Sira Ferradans, Gui-Song Xia, Gabriel Peyré, Jean–François Aujol2011

TL;DR

This paper introduces a novel method for mixing and synthesizing textures by leveraging optimal transport theory to define geodesic paths between Gaussian texture models. This approach allows for the creation of new, visually coherent textures that blend the characteristics of existing ones, extending to both grayscale and color images.

What Problem Does This Paper Address?

Synthesizing realistic textures is a fundamental challenge in computer graphics and image processing. While various methods exist, creating new textures that seamlessly blend the features of multiple existing textures (texture mixing) is particularly difficult, especially when dealing with complex statistical properties. Existing approaches often rely on mixture models or non-parametric histogram averaging, which can be computationally intensive or struggle with certain texture types. This paper addresses the need for a more flexible and efficient method to mix textures, specifically focusing on stationary Gaussian texture models, by defining a principled way to interpolate between them.

Methods

The core of this paper lies in applying optimal transport (OT) theory to Gaussian texture models. The authors first define a "spot noise" model, a specific type of stationary Gaussian process, which can be learned from input textures. They then leverage the L2 optimal transport distance to define a geodesic path between two such Gaussian models. This geodesic path allows for the generation of intermediate texture models. For the case of spot noise models, they derive a closed-form solution for this geodesic path, ensuring that the resulting interpolated model is also a spot noise model. Furthermore, they extend this concept to compute the optimal transport barycenter of multiple Gaussian models, providing a way to mix more than two textures. The synthesis of new textures is achieved by sampling from these interpolated Gaussian models.

Datasets

The paper uses "original textures" as input for their experiments. Specific datasets are not explicitly named or introduced as new benchmarks. The examples shown in the figures (Fig. 1, Fig. 2) consist of various natural textures, including grass, water, wood, and fabric patterns.

Key Insights and Ideas

  • Optimal Transport as a Texture Interpolator: The paper demonstrates that optimal transport, typically used for comparing probability distributions, can be effectively repurposed to define meaningful interpolations between texture models. This provides a principled way to blend texture characteristics.
  • Geodesic Paths for Texture Synthesis: By defining geodesic paths in the space of Gaussian texture models, the authors can generate a continuous spectrum of intermediate textures that smoothly transition between input textures. This is a significant improvement over ad-hoc mixing methods.
  • Spot Noise Models and Tractable Geodesics: The focus on stationary Gaussian "spot noise" models is crucial. It allows for a tractable derivation of the geodesic path, ensuring that the interpolated models remain within the same class of spot noise models, which simplifies synthesis.
  • Barycenters for Multi-Texture Mixing: The extension to computing the barycenter of multiple Gaussian models provides a powerful tool for mixing an arbitrary number of textures, generalizing the pairwise mixing approach.

Key Results

  • The method successfully generates new textures that exhibit visual features from the original input textures, demonstrating effective texture mixing.
  • The synthesized textures maintain the characteristics of the original ones, even when interpolating between them.
  • The method is capable of reproducing small periodic patterns within the synthesized textures.
  • The paper shows qualitative results of mixing two and three input textures, illustrating the effectiveness of both the geodesic path and barycenter approaches.

Limitations and Future Work

  • Rank-Deficient Covariances: The paper notes that for rank-1 covariances (spot noise models), if the leading eigenvectors of the covariances are not orthogonal, there can be an infinite number of geodesic paths. While they provide a specific solution for spot noise models, the general case of rank-deficient covariances might still pose challenges.
  • Uniqueness of Barycenters: The theoretical analysis of the uniqueness of the barycenter when all input covariances are rank-deficient is left for future work.
  • Color Texture Extension: While the paper mentions extending the spot noise model to color textures, the numerical examples and detailed derivations primarily focus on the mathematical framework for Gaussian distributions, with specific emphasis on the spot noise case. The full implications and performance for d=3 color textures might require further exploration.
  • Computational Complexity: While the method is described as leading to "fast numerical schemes," the paper does not provide explicit computational complexity analysis for the barycenter computation, particularly for color images where matrix square roots of 3x3 matrices are involved.

Summary updated

Abstract

Let us consider the modeling of textures f ∈ R N×d, where N = N1 × N2 is the number of pixels in the image, and d is the number of channels (d = 1 for grayscale and d = 3 for color datasets). We define a Gaussian distribution as µ = N (m, Σ) where m ∈ R N×d is the mean of the distribution and Σ ∈ R Nd×Nd is a positive semi-definite covariance matrix. We focus on stationary random fields X ∼ µ, which means that the distributions of X and X( · + τ) are the same for any translation vector τ ∈ Z 2. For simplicity of the exhal-00662720,
BibTeX
@misc{ferradans2011optimal,
  author = {Sira Ferradans and Gui-Song Xia and Gabriel Peyré and Jean–François Aujol},
  title = {{Optimal Transport Mixing of Gaussian Texture Models}},
  year = {2011},
}

Loading related content...

Figure