Probabilistic Curve Learning: Coulomb Repulsion and the Electrostatic Gaussian Process

Ye Wang, David B. DunsonNeurIPS 2015

The Problem with Manifold Learning

In machine learning, we often assume that high-dimensional data—such as video frames or complex sensor arrays—actually resides on a much simpler, low-dimensional "manifold." Uncovering this underlying structure is the primary goal of manifold learning.

While probabilistic methods like the Gaussian Process Latent Variable Model (GP-LVM) are theoretically appealing because they naturally handle noise and uncertainty, they suffer from a significant identifiability issue. In standard GP-LVM, latent coordinates are typically drawn from simple uniform or Gaussian distributions. Because the model is highly flexible, the optimization process often "cheats" by clumping all the latent points into a tiny corner of the space. This clumping creates "holes" in the learned manifold, leading to erratic mapping functions and poor performance on downstream tasks like denoising or predicting missing data.

The Electrostatic Gaussian Process (electroGP)

To solve this clumping issue, the authors introduce the Electrostatic Gaussian Process (electroGP), a novel probabilistic framework that incorporates a physics-inspired Coulomb Repulsive Process (Corp) as a prior.

Inspired by electrostatic physics, the Corp prior treats latent points like electrically charged particles that repel one another. The probability of two points being near each other shrinks exponentially as the distance between them decreases. The authors utilize a periodic sine-based distance function to ensure the "repulsive field" is homogeneous and free of edge effects.

Effect of the Coulomb Repulsive Process (Corp) prior on latent point distribution.

This "electrostatic" repulsion acts as a soft regularizer that naturally fills holes in the latent space, ensuring the learned curve is smooth and continuous across the entire data range. A fascinating property of the Corp prior is its "self-truncation": the log-likelihood drops to $-\infty$ if two points overlap. This means that once an initial ordering is established (e.g., via Locally Linear Embedding), the optimizer will never "jump" points over each other, maintaining a stable topological structure throughout training.

Modeling Complex Curves and Uncertainty

The electroGP model combines the Corp prior for latent locations with a Gaussian Process (GP) prior for the mapping function that translates those locations back into high-dimensional data. Unlike traditional "Principal Curves"—which can be highly unstable and tend to "cut corners" in high-curvature regions—electroGP leverages the GP's inherent smoothness and the Corp's spread to snake accurately through complex data densities.

electroGP fitting a complex 3D spiral dataset with uncertainty bands.

Furthermore, the authors propose a heuristic algorithm to generate "uncertainty bands" around the learned curve. By sampling from the posterior predictive distribution and calculating quantiles of the shortest distances to the mean curve, they provide a visual and mathematical measure of how confident the model is about the manifold's exact shape.

Experimental Results

The model was evaluated on synthetic 2D/3D distributions (like spirals and parabolas) as well as real-world high-dimensional video datasets, demonstrating significant improvements over baselines:

  • Video Inpainting (Rotating Teapot): On a dataset of 200 RGB frames of a rotating teapot, 10 frames were withheld as "un-ordered" and partially corrupted. electroGP achieved a Mean Square Error (MSE) of 70.62, vastly outperforming Bayesian GP-LVM (MSE 450.75). Visually, electroGP reconstructed fine details, like the teapot handle, that other models blurred or lost entirely.
  • Image Denoising: Using 100 grayscale frames of a shrinking shockwave heavily corrupted by white noise, electroGP achieved a denoising MSE of $1.8 \times 10^{-3}$. Standard image filters like Non-Local Means (NLM) and Isotropic Diffusion (IsD) performed significantly worse, hovering around an MSE of 61-63.
  • Missing Data Imputation: In the same shockwave sequence, the model perfectly regenerated 5 entirely missing consecutive frames. The MSE for the missing frame reconstruction was an impressive $2 \times 10^{-5}$, compared to 13 for standard linear interpolation.

Limitations and Future Work

The primary limitation of the current work is its exclusive focus on one-dimensional manifolds (curves). While the authors provide a mathematical definition for a multivariate (higher-dimensional) Corp, it is not implemented or tested in this paper.

Future research directions include developing efficient algorithms to apply electroGP to multidimensional manifolds (surfaces and volumes), automating the process of learning the intrinsic dimensionality ($p$) of the manifold, and improving the computational efficiency of the repulsive calculations, which could become a bottleneck as the number of data points grows.

Summary updated

BibTeX
@inproceedings{wang2015probabilistic,
  author = {Ye Wang and David B. Dunson},
  title = {{Probabilistic Curve Learning: Coulomb Repulsion and the Electrostatic Gaussian Process}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2015},
}
Figure