Introduction
Human Pose Estimation (HPE) has seen significant progress, yet most current 3D methods remain viewpoint-dependent. While they excel at front-facing imagery, performance degrades sharply when cameras are positioned at unconventional angles, such as top-down views. This limitation stems from standard Convolutional Neural Networks (CNNs) prioritizing viewpoint invariance—identifying an object regardless of its orientation—rather than viewpoint equivariance, where internal representations transform predictably with the camera angle.
The DECA Architecture
To address this, we present DECA (Deep viewpoint-Equivariant Capsule Autoencoder). Unlike traditional scalar-based neurons, DECA utilizes Matrix Capsules to represent human joints as $4 \times 4$ pose matrices. This structure explicitly encodes the geometric relationship between body parts.
The architecture employs Variational Bayes (VB) Routing to establish part-whole relationships (e.g., hand to elbow) independently of the viewpoint. Furthermore, the model is trained with multi-task decoders for 3D pose, 2D pose, and depth map reconstruction, alongside a novel Inverse Graphics Loss. This loss ensures the learned capsule weights are invertible, effectively "de-rendering" the input image into its geometric components.
Experimental Evaluation
DECA was evaluated on the ITOP and PanopTOP31K datasets, which provide both front and top-down perspectives. The primary challenge was "viewpoint transfer": training on one view and testing on another.
Key Results
DECA achieves state-of-the-art performance on depth-based HPE, reaching 88.75 mAP on ITOP front-view and 86.92 mAP on top-view. Most notably, in the viewpoint transfer test (training on front, testing on top), DECA achieved 51.85 mAP, more than doubling the performance of the best baseline (20.40 mAP).
Conclusion
By prioritizing viewpoint equivariance over invariance, DECA demonstrates that capsule-based architectures can internalize the "geometrical logic" of the human body. This allows for robust generalization to unseen camera angles with significantly higher data efficiency than traditional CNN-based approaches. Future work will focus on extending this framework to fine-grained tasks like hand pose estimation and testing on larger "in-the-wild" datasets.
Summary updated