Implicitly Constrained Gaussian Process Regression for Monocular Non-Rigid Pose Estimation

Mathieu Salzmann, Raquel UrtasunNeurIPS 2010 · OpenReview

The Challenge of 3D Pose Estimation

Estimating the 3D pose of a human or a deforming surface from a single 2D image is an inherently "ill-posed" problem. Because a single 2D projection can correspond to an infinite number of 3D configurations, researchers must rely on physical constraints to narrow down the possibilities. For example, a person's bone lengths remain constant as they move, and rotation matrices must remain orthonormal.

Historically, approaches to this problem fall into two categories:

  1. Generative models: These are highly accurate but require a precise initial guess to avoid getting trapped in local minima during optimization.
  2. Discriminative predictors: These models learn a direct mapping from 2D images to 3D poses. While they are fast, they often treat different components of the pose independently. This can lead to physically impossible results, such as unnaturally stretching limbs.

Previous attempts to fix the physical violations in discriminative models involved adding a "correction" step during inference. However, this requires solving a complex, non-convex optimization problem for every single frame, making it far too slow for real-time applications.

Methodology: Implicitly Constrained GP Regression

To solve the speed-versus-accuracy dilemma, the authors introduce a method to bake physical constraints directly into Gaussian Process (GP) regression. Their approach relies on two key mathematical insights:

1. The "Linear for Free" Property

The authors demonstrate that if your training data satisfies a set of linear constraints (e.g., $Ay = b$), the mean prediction of a Gaussian Process trained on that data will automatically satisfy those same constraints. This occurs because the GP mean is fundamentally a linear combination of the training examples.

2. Linearizing Quadratic Constraints

Unfortunately, most physical constraints in pose estimation are quadratic, not linear (such as the $L_2$-norm of a quaternion or the fixed distance between two joints). To handle this, the authors utilize a clever change of variables.

Instead of regressing the 3D pose $Z$ directly, the model regresses the matrix $Q = Z^T Z$.

  • In this transformed space, quadratic constraints on the original pose $Z$ become simple linear constraints on $Q$.
  • The GP predicts the elements of $Q$, implicitly satisfying the constraints without extra optimization.
  • To recover the original 3D pose $Z$, the system uses Singular Value Decomposition (SVD) to factorize the predicted matrix.

By moving the constraint enforcement into the data representation itself, the authors completely bypass the need for iterative, heavy solvers at test time.

Datasets and Evaluation

The authors evaluated their implicitly constrained GP on several diverse tasks to prove its versatility:

Examples of the evaluation datasets, including rigid shapes, deformable meshes, and human pose sequences.
  • Rigid Rotation: Estimating the rotation of a 3D square plane using quaternions and rotation matrices.
  • Surface Deformations: Reconstructing inextensible meshes (both $2 \times 2$ and $9 \times 9$) using synthetic data and real paper video sequences.
  • Human Pose Estimation: Tracking human movement using the HumanEva dataset, a standard benchmark containing synchronized multi-view video and motion capture data.

Key Results

Treating constraints as a fundamental property of the data distribution yielded significant improvements across the board:

  • Accuracy: The proposed method consistently outperformed standard GPs and specialized algorithms like the "Perspective-n-Point" (PnP) solution, showing particular robustness in the presence of image noise.
  • Constraint Satisfaction: While standard GPs exhibited significant physical violations (e.g., predicting limbs that stretched by 10-20%), the implicitly constrained method kept violations near zero.
  • Speed: This is the most dramatic improvement. For a $9 \times 9$ mesh, the previous state-of-the-art method (which enforced constraints at inference) took about 1.6 seconds per frame. The proposed method took only 9 milliseconds—a speedup of nearly 180x.

Limitations and Future Directions

While highly effective, the method does have a few limitations. The recovery step via SVD assumes the predicted matrix has a specific rank. If the GP prediction is noisy, the factorization might lose some information, though experiments showed this loss is minimal in practice. Additionally, the SVD recovery step introduces sign and rotation ambiguities (e.g., determining if an object is facing toward or away from the camera), which the authors currently resolve using a secondary PnP step or image correspondences.

Future work may extend this framework to include temporal dynamics—enforcing constraints on how a pose changes over time—and exploring richer image features to further disambiguate complex 3D geometry.

Summary updated

Abstract

Estimating 3D pose from monocular images is a highly ambiguous problem. Physical constraints can be exploited to restrict the space of feasible configurations. In this paper we propose an approach to constraining the prediction of a discriminative predictor. We first show that the mean prediction of a Gaussian process implicitly satisfies linear constraints if those constraints are satisfied by the training examples. We then show how, by performing a change of variables, a GP can be forced to satisfy quadratic constraints. As evidenced by the experiments, our method outperforms state-of-the-art approaches on the tasks of rigid and non-rigid pose estimation.
BibTeX
@inproceedings{salzmann2010implicitly,
  author = {Mathieu Salzmann and Raquel Urtasun},
  title = {{Implicitly Constrained Gaussian Process Regression for Monocular Non-Rigid Pose Estimation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2010},
  url = {https://openreview.net/forum?id=BJE6ytbuWS},
}
Figure