DECA: Deep viewpoint-Equivariant human pose estimation using Capsule Autoencoders

Nicola Garau, Niccolò Bisagno, Piotr Bródka, Nicola ConciICCV 2021 · DOI · arXiv

Introduction

Human Pose Estimation (HPE) has seen significant progress, yet most current 3D methods remain viewpoint-dependent. While they excel at front-facing imagery, performance degrades sharply when cameras are positioned at unconventional angles, such as top-down views. This limitation stems from standard Convolutional Neural Networks (CNNs) prioritizing viewpoint invariance—identifying an object regardless of its orientation—rather than viewpoint equivariance, where internal representations transform predictably with the camera angle.

The DECA Architecture

To address this, we present DECA (Deep viewpoint-Equivariant Capsule Autoencoder). Unlike traditional scalar-based neurons, DECA utilizes Matrix Capsules to represent human joints as $4 \times 4$ pose matrices. This structure explicitly encodes the geometric relationship between body parts.

The architecture employs Variational Bayes (VB) Routing to establish part-whole relationships (e.g., hand to elbow) independently of the viewpoint. Furthermore, the model is trained with multi-task decoders for 3D pose, 2D pose, and depth map reconstruction, alongside a novel Inverse Graphics Loss. This loss ensures the learned capsule weights are invertible, effectively "de-rendering" the input image into its geometric components.

DECA outputs showing 3D skeleton and depth map reconstruction across front and top views.

Experimental Evaluation

DECA was evaluated on the ITOP and PanopTOP31K datasets, which provide both front and top-down perspectives. The primary challenge was "viewpoint transfer": training on one view and testing on another.

Key Results

DECA achieves state-of-the-art performance on depth-based HPE, reaching 88.75 mAP on ITOP front-view and 86.92 mAP on top-view. Most notably, in the viewpoint transfer test (training on front, testing on top), DECA achieved 51.85 mAP, more than doubling the performance of the best baseline (20.40 mAP).

Qualitative comparison of DECA (Ours) against 2D/3D HPE and 3D HMR baselines. DECA maintains structural integrity across both front and top-down viewpoints.

Conclusion

By prioritizing viewpoint equivariance over invariance, DECA demonstrates that capsule-based architectures can internalize the "geometrical logic" of the human body. This allows for robust generalization to unseen camera angles with significantly higher data efficiency than traditional CNN-based approaches. Future work will focus on extending this framework to fine-grained tasks like hand pose estimation and testing on larger "in-the-wild" datasets.

Summary updated

BibTeX
@inproceedings{garau2021deca,
  author = {Nicola Garau and Niccolò Bisagno and Piotr Bródka and Nicola Conci},
  title = {{DECA: Deep viewpoint-Equivariant human pose estimation using Capsule Autoencoders}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01147},
}
Figure