Canonical Pose Reconstruction from Single Depth Image for 3D Non-rigid Pose Recovery on Limited Datasets

Fahd Alhamazani, Yu-Kun Lai, Paul L. RosinarXiv 2025 · arXiv

TL;DR

Reconstructing 3D non-rigid shapes like humans or animals from a single depth image typically requires massive datasets to cover every possible pose. This paper introduces a novel two-stage deep learning framework that first normalizes a single-view depth image into a standardized "canonical" pose before reconstructing the full 3D voxelized shape. Remarkably, this approach achieves state-of-the-art reconstruction accuracy using a fraction of the data—only about 300 training samples.

What Problem Does This Paper Address?

3D shape reconstruction from 2D or partial inputs is a cornerstone of computer vision, but non-rigid objects (such as a bending human or a running animal) complicate things immensely. Because these objects can deform in infinite ways, deep learning models usually require massive datasets to learn the entire "deformation space."

To simplify this, researchers often try to map shapes back to a default, standardized pose called a canonical form (like a human T-pose). However, existing canonical-form methods suffer from two major flaws:

  1. They typically assume the input is already a complete 3D mesh, making them useless for partial, single-view depth images.
  2. They often distort the shape's appearance because they rely strictly on basic geometric distance metrics.

This paper tackles the challenge of recovering both a canonical depth image and a full 3D shape from a single, incomplete depth image, even when training data is extremely limited.

Methods

The authors propose a two-stage pipeline to decouple pose normalization from 3D shape completion.

Figure 1: Stage one, overview of model. where the input depth image in any pose and output canonicalised depth image

Stage One: Canonical Pose Depth Reconstruction

Given a single depth image and its foreground mask, the model estimates what that object would look like in its canonical pose. This is achieved through three sub-components:

  • Local Feature Extractor (LFE): An encoder-decoder network that extracts high-dimensional local features from the input image and mask.
  • Multi-Scale Feature Extractor (MSF): To resolve long-range pixel dependencies (crucial for "un-bending" limbs), the model uses three parallel encoders with distinct dilation rates (1, 2, and 3). These multi-scale features are concatenated and mapped through a fusing decoder.
  • Canonical Reconstruction Component: Combines the local and multi-scale features to predict the canonical depth image and mask.

Stage Two: 3D Shape Reconstruction & Pose Recovery

Once the canonical depth image is estimated, Stage Two lifts the 2D data into 3D space.

Stage Two Architecture for 3D Shape Reconstruction

The architecture uses a Generative Adversarial Network (GAN) framework featuring:

  • A Shape Encoder that processes the predicted canonical depth image to capture structural details without occlusion.
  • A Pose Encoder that processes the original, deformed depth image to capture the input pose.
  • A Decoder that fuses these embeddings to output a high-resolution voxelized 3D shape ($256 \times 256 \times 256$).
  • A WGAN-GP Discriminator customized to output a vector rather than a scalar, which stabilizes training and smooths the reconstructed surface.

Datasets

The model was evaluated on three distinct non-rigid datasets:

  1. Synthetic Human Dataset: 300 shapes across 15 subjects (males, females, and children) with 20 poses each.
  2. Real Human Dataset (CAESAR): 40 subjects (20 males, 20 females) captured in 10 different poses.
  3. TOSCA Dataset: 80 objects featuring highly varied animal and human shapes (cats, dogs, wolves, horses, centaurs, gorillas) with varying pose counts.

Key Insights and Ideas

  • Decoupling Pose from Geometry: By transforming the partial input into a rigid canonical form first, the model can apply simpler rigid-object reconstruction techniques in the second stage. This drastically reduces the complexity of the learning task.
  • Extreme Data Efficiency: While standard implicit-function or transformer-based reconstruction models require thousands of diverse models to generalize, this architecture achieves high fidelity with only ~300 training samples.
  • The Power of Dilated Convolutions for Deformations: Standard convolutions struggle with the long-range pixel shifts inherent to pose changes. Parallel dilated encoders effectively act as a coordinate-shifting mechanism, allowing the network to understand where a hand or leg should move to reach the T-pose.

Key Results

The framework was compared against both traditional geometric methods (for canonical pose retrieval) and modern deep learning methods like ShapeFormer and IF-Net (for 3D reconstruction).

Stage One (Canonicalization & Retrieval)

On the Synthetic Human dataset, the proposed method achieved a Nearest Neighbor (NN) retrieval score of 0.51, vastly outperforming classical methods like Least Squares MDS (0.01) and GPS (0.40). On the diverse TOSCA dataset, it achieved an NN score of 0.91 (compared to Constrained MDS at 0.88).

Stage Two (3D Reconstruction)

The model demonstrated superior shape completion performance using Intersection over Union (IoU) metrics:

  • Synthetic Dataset: Achieved an IoU of 0.81, outperforming ShapeFormer (0.65) and IF-Net (0.69).
  • TOSCA Dataset: Achieved an IoU of 0.68, outperforming ShapeFormer (0.57) and IF-Net (0.56).
Qualitative results showing successful 3D reconstruction from a single input view

Limitations and Future Work

Despite its strong performance on limited data, the authors note a few areas for improvement:

  • Domain Gap: When trained on synthetic data and tested on real human data (CAESAR), the retrieval performance dropped slightly (ranking second on the NN metric), indicating sensitivity to the synthetic-to-real domain gap.
  • Hardware Constraints: Due to GPU memory limits, the authors could not scale beyond three parallel encoders in the Multi-Scale Feature Extractor.
  • Voxel Resolution: The model outputs a $256^3$ voxel grid. While high-resolution for voxels, modern implicit representations (like Neural Radiance Fields or Implicit Functions) could offer finer surface details without the cubic memory scaling of voxels.

Summary updated

Abstract

3D reconstruction from 2D inputs, especially for non-rigid objects like humans, presents unique challenges due to the significant range of possible deformations. Traditional methods often struggle with non-rigid shapes, which require extensive training data to cover the entire deformation space. This study addresses these limitations by proposing a canonical pose reconstruction model that transforms single-view depth images of deformable shapes into a canonical form. This alignment facilitates shape reconstruction by enabling the application of rigid object reconstruction techniques, and supports recovering the input pose in voxel representation as part of the reconstruction task, utilizing both the original and deformed depth images. Notably, our model achieves effective results with only a small dataset of approximately 300 samples. Experimental results on animal and human datasets demonstrate that our model outperforms other state-of-the-art methods.
BibTeX
@misc{alhamazani2025canonical,
  author = {Fahd Alhamazani and Yu-Kun Lai and Paul L. Rosin},
  title = {{Canonical Pose Reconstruction from Single Depth Image for 3D Non-rigid Pose Recovery on Limited Datasets}},
  year = {2025},
  eprint = {2505.17992},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.17992},
}
Figure