Abstract
We present a probabilistic structured prediction method for learning input-output dependencies where correlations between outputs are modeled as low-dimensional manifolds constrained by both geometric structure and predictive utility. The Supervised Spectral Latent Variable Model (SSLVM) jointly optimizes a low-dimensional latent manifold to preserve output distances while maximizing predictive accuracy. By enforcing these geometric constraints, the model ensures that the discovered manifold is specifically tailored for the task at hand. We demonstrate the effectiveness of SSLVM on 3D human pose estimation from 2D silhouettes, showing significant improvements over unsupervised manifold learning techniques.
Introduction
In many computer vision tasks, such as 3D human pose estimation, researchers must map high-dimensional inputs (e.g., image silhouettes) to high-dimensional, highly correlated outputs (e.g., 3D joint positions). This is a "structured prediction" problem. Traditional approaches often rely on unsupervised manifold learning to reduce dimensionality before learning a mapping. However, a manifold optimized for data representation (unsupervised) is rarely the optimal manifold for prediction (supervised). Furthermore, separate training of the input-to-manifold and manifold-to-output components often leads to uncalibrated noise models and suboptimal end-to-end performance.
Methods
The SSLVM addresses these issues by treating the manifold as a hidden layer in a joint probabilistic model:
$$p(y, x|r) = p(y|x)p(x|r)$$
where $r$ represents the input, $y$ the output, and $x$ the latent variable.
Geometric Constraints and Optimization
A key innovation of SSLVM is the inclusion of implicit geometric constraints. A penalty term ensures that distances between points in the latent space $x$ reflect the distances between their corresponding points in the output space $y$. This allows the model to leverage the strengths of spectral methods—such as PCA, ISOMAP, or Laplacian Eigenmaps—within a supervised framework.
Training involves a two-stage iterative process:
- Input-to-Latent Optimization: Optimizing the predictor that maps inputs to the latent space.
- Latent-to-Output Optimization: Using Expectation-Maximization (EM) to update the mapping from the latent space to the output while enforcing geometric constraints.
To handle the inherent ambiguity in tasks like 3D pose estimation, the model employs probabilistic mappings such as Bayesian Mixtures of Experts (BME) and Relevance Vector Machines (RVM).
Experimental Evaluation
The model was evaluated using the HumanEva dataset, a benchmark for human motion analysis containing synchronized video and 3D motion capture data. The experiments utilized approximately 6,000 training and 5,800 test samples. Inputs consisted of 270-dimensional silhouette descriptors, while outputs were 60-dimensional vectors of 3D body joint positions.
Results
SSLVM consistently outperformed baseline methods, including standard RVM and unsupervised SLVM. In 3D pose reconstruction, it achieved an average joint error reduction of approximately 10mm compared to models without joint manifold optimization.
For complex datasets involving multiple activities (walking, jogging, boxing), SSLVM with non-linear constraints (e.g., ISOMAP) showed even greater gains. For instance, on mixed motion sequences, SSLVM reduced error from ~78mm (RVM) to ~61mm (SSLVM-LE). Qualitative analysis revealed that SSLVM's latent spaces exhibit higher variance than unsupervised PCA, effectively spreading out data to better account for uncertainty in image-based predictors.
Conclusion
The Supervised Spectral Latent Variable Model demonstrates that joint calibration of the input-to-latent and latent-to-output mappings is crucial for high-performance structured prediction. By forcing the latent manifold to preserve the structure relevant to the output task, SSLVM achieves superior accuracy and robustness. Future work will explore incorporating temporal dependencies and extending the framework to semi-supervised settings.
Summary updated