Abstract
Accurate Head and Body Pose Estimation (HBPE) is critical for understanding social interactions but remains challenging in crowded, low-resolution environments where manual labeling is expensive. This work introduces a transductive matrix completion framework that replaces traditional local Laplacian smoothing with Gaussian Process Regression (GPR). By capturing global temporal trends and incorporating head-body orientation coupling, the proposed method significantly outperforms existing baselines, maintaining high accuracy even with extremely sparse (5%) labeling.
Introduction
Head and Body Pose Estimation (HBPE) serves as a cornerstone for social signal processing, providing essential cues about attention and group dynamics. However, performing HBPE in "the wild"—such as in surveillance footage of crowded social events—is fraught with difficulties including low resolution, frequent occlusions, and poor lighting.
While deep learning has achieved remarkable success in high-resolution, frontal-view scenarios, it often struggles in these sparse, noisy environments where large-scale training data is unavailable. Annotating every frame of a video is prohibitively expensive. Previous matrix completion approaches attempted to fill missing labels by treating the video as a grid, but they relied heavily on Laplacian smoothing. This local approach often fails to capture long-term temporal trends when labels are sparse. This paper addresses these limitations by introducing a global temporal interpolation strategy.
Methodology
The proposed transductive learning model is built upon three primary pillars designed to handle data sparsity and physical constraints:
- Matrix Completion & Nuclear Norm Minimization: The system constructs a heterogeneous matrix concatenating visual features (HOG descriptors) with pose labels. By minimizing the nuclear norm, the model recovers the underlying low-dimensional relationship between features and poses from partial data.
- Gaussian Process Regression (GPR) for Temporal Smoothing: Replacing simple local smoothing, GPR serves as the core interpolation mechanism. By considering the global context of the time series, GPR recovers non-linear patterns and long-term trends that local methods overlook.
- Head-Body Coupling: To reflect physical reality, the model includes a regularization term that penalizes extreme discrepancies between head and body orientations while maintaining enough flexibility to capture natural movement.
The resulting optimization problem is solved iteratively using the Alternating Direction Method of Multipliers (ADMM).
Experimental Setup
The model was evaluated using the SALSA dataset, which captures 18 individuals interacting during a poster session and a cocktail party. The setup utilized a 4-view surveillance camera network. Ground truth annotations for head and body poses were provided every 3 seconds across eight discrete classes (N, NE, E, SE, S, SW, W, NW). Visual features were represented by 100-dimensional vectors derived from Histogram of Gradients (HOG) and Principal Component Analysis (PCA).
Key Results and Insights
The GPR-based method demonstrated a drastic improvement over the previous state-of-the-art (MC-HBPE), particularly in high-sparsity regimes:
- Performance at 5% Sparsity: Even with only 5% of labels available, the model achieved approximately 62% accuracy for head pose and 70% for body pose. The baseline method frequently failed to converge or produce meaningful results at this level of sparsity.
- Stability: The GPR approach showed significantly tighter error bars, indicating higher stability across different participants and movement styles.
- Entropy of Movement: Information entropy analysis confirmed that head pose is fundamentally more difficult to predict than body pose due to higher movement frequency and diversity. This explains the consistent performance gap between head and body estimation across all tested models.
Conclusion
By shifting from local averaging to global functional fitting via Gaussian Process Regression, this framework provides a robust solution for pose estimation in data-starved environments. While the current transductive nature and $O(n^3)$ complexity of GPR present scaling challenges, the results set a new benchmark for sparse HBPE. Future work will explore replacing hand-crafted features with deep CNN features and integrating multimodal data from wearable sensors to further refine the matrix completion process.
Summary updated