Abstract
Multi-view 3D Human Pose Estimation (HPE) often struggles to generalize to unseen camera configurations, particularly in "camera-parameter-free" settings. We present PoseIRM, a framework that leverages Invariant Risk Minimization (IRM) and novel 2D pose synthesis strategies to learn camera-invariant features. By training across thousands of simulated environments, PoseIRM achieves state-of-the-art performance on standard benchmarks and demonstrates significantly improved robustness in uncalibrated, real-world scenarios.
Introduction
Multi-view 3D HPE typically requires precise camera calibration to triangulate human joints. While "camera-parameter-free" methods offer greater flexibility for "in-the-wild" applications, they frequently suffer from a significant generalization gap. Due to the limited number of fixed camera views in standard datasets like Human3.6M, models often "memorize" specific viewpoints rather than learning underlying geometric principles. Consequently, performance degrades sharply when these models are deployed in novel camera setups. PoseIRM addresses this by treating generalization as a domain generalization problem, forcing the model to ignore spurious camera-specific features.
Methods
PoseIRM introduces two primary components to enhance generalization:
Pose Synthesis Algorithms (PSA)
To provide the diverse environments necessary for robust training, the authors synthesize 2D-3D pose pairs by projecting ground-truth 3D poses into thousands of virtual camera views. Two variants are proposed:
- DAPSA (Distribution-Aware PSA): Models 2D detector noise using a Student’s t-distribution to simulate real-world detection inaccuracies.
- MVCPSA (Multi-view Consistent PSA): Ensures geometric consistency of noise across views, mimicking the systematic failure modes of actual 2D detectors.
Invariant Risk Minimization (IRM)
The core of the framework is the application of IRM. Unlike standard empirical risk minimization which averages error across all data, IRM seeks a representation that is simultaneously optimal across all training environments (where each camera setting constitutes an environment). This encourages the network to discard "spurious" features—such as specific camera heights or angles—and focus on "invariant" features related to human skeletal structure.
The architecture employs a PoseFormer backbone for feature extraction and a transformer decoder for multi-view fusion, outputting both 3D poses and estimated camera parameters.
Experimental Results
The model was evaluated on Human3.6M and TotalCapture datasets, supplemented by over 15,000 unique synthetic camera settings.
Performance and Generalization
PoseIRM achieved a state-of-the-art Mean Per Joint Position Error (MPJPE) of 25.1 mm on Human3.6M, surpassing previous parameter-free methods. More importantly, on the TotalCapture dataset's unseen settings, PoseIRM reduced error from 30.7 mm to 16.6 mm, demonstrating a massive leap in robustness.
Camera Parameter Estimation
Beyond pose estimation, the model significantly improved camera orientation regression, reducing rotation error to approximately 1°–2°, a substantial improvement over the 6° error seen in competing models.
Conclusion
PoseIRM demonstrates that the generalization gap in camera-parameter-free 3D HPE is largely a result of viewpoint memorization. By combining large-scale coordinate-based synthesis with IRM, the framework successfully learns geometric invariants. While the model remains dependent on 2D detector quality and requires careful tuning of the IRM penalty, it sets a new standard for robust, uncalibrated 3D human pose estimation. Future work may extend this approach to dynamic, frame-by-frame camera movements.
Summary updated