DualPM: Dual Posed-Canonical Point Maps for 3D Shape and Pose Reconstruction

Ben Kaye, Tomas Jakab, Shangzhe Wu, Christian Rupprecht, Andrea Vedaldi

TL;DR

DualPM introduces a novel representation for 3D reconstruction that maps image pixels to two distinct 3D points: one in the observed camera space and one in a neutral "canonical" pose. By predicting these "Dual Point Maps," the model can simultaneously recover the full 3D shape and articulated pose of deformable objects—even hidden parts—while training exclusively on minimal synthetic data.

What Problem Does This Paper Address?

Reconstructing the 3D shape and pose of deformable objects (like animals) from a single 2D image is a classic "ill-posed" problem in computer vision. While recent breakthroughs like DUSt3R have simplified static scene reconstruction by predicting "point maps" (mapping pixels to 3D coordinates), deformable objects add a layer of complexity: articulation.

Traditional methods often try to regress bone rotations for a skeleton or fit a category-level template. However, these approaches struggle with the non-linear nature of 3D rotations and often fail to generalize to unusual poses or shapes not seen during training. Furthermore, standard point maps only capture the visible surface, leaving the "back" of the object a mystery.

Methods

The authors propose Dual Point Maps (DualPM). Instead of predicting one 3D coordinate per pixel, the network predicts two:

  1. Posed Point Map ($P$): The 3D location of the point in the camera's reference frame (the "where is it now" map).
  2. Canonical Point Map ($Q$): The 3D location of that same point on a "neutral" version of the object (the "where would it be if it were standing still" map).

As shown in Figure 1, the difference between these two maps ($P - Q$) effectively defines the deformation field, making pose estimation a simple subtraction problem.

Figure 1. Left: We map an image of an object to its Dual Point Maps (DualPMs), a pair of point maps P, defined in a camera space, and Q, defined in a canonical space where the object has a neutral pose. The pose is thus given by the flow P −Q. Right: The DualPMs are easy to predict with a neural network, enabling effective 3D object reconstruction and facilitating geometric tasks like detecting 3D keypoints and fitting a 3D skeleton. For visualization, we color each point with its coordinate in the canonical point maps.

To handle the "missing" parts of the object, the authors introduce Amodal Point Maps. Inspired by "depth peeling" in computer graphics, the model predicts multiple layers of points for each pixel. If a camera ray passes through a horse's front leg and then its torso, the model predicts coordinates for both intersection points. This is managed via a layered representation and associated opacity maps (Figure 2).

Figure 2. An amodal point map associates each pixel u to an even number of objects points pi, corresponding to the locations where the pixel’s ray intersects the object’s surface. Predicting an amodal point map reconstructs the entire object despite self-occlusions.

The architecture uses a pre-trained feature extractor (combining DINOv2 and Stable Diffusion features) to first predict the canonical map $Q$, and then uses $Q$ as a feature to predict the posed map $P$. This conditioning helps the model generalize better to real-world images.

Datasets

  • Training: The models are trained purely on synthetic data generated from just one or two rigged 3D models per category (e.g., one horse model) from the Animodel dataset.
  • Evaluation:
    • PASCAL VOC: Used for real-world qualitative results and quantitative keypoint transfer tasks.
  • Animodel-Points: A new, more robust benchmark introduced by the authors to evaluate point cloud reconstruction accuracy using bi-directional Chamfer Distance.

Key Insights and Ideas

  • Pixel-Labeling is Easier: The authors argue that predicting pixel-aligned 3D coordinates is a much simpler task for a neural network than regressing a chain of abstract 3D bone rotations. It turns a complex global optimization problem into a dense prediction task.
  • Canonical Invariance: The canonical map $Q$ is viewpoint- and pose-invariant. This makes it a very stable intermediate representation that simplifies the subsequent prediction of the posed map $P$.
  • Synthetic-to-Real Generalization: By focusing on geometric point maps rather than raw pixels, the model generalizes remarkably well from a single synthetic 3D asset to diverse real-world photos of different breeds and ages (e.g., training on an adult horse but succeeding on a foal).

Key Results

DualPM significantly outperforms existing state-of-the-art methods like 3D-Fauna across multiple metrics:

  • Keypoint Transfer: On PASCAL VOC, DualPM achieved a 73.2% [email protected] for horses, compared to 53.9% for 3D-Fauna.
  • Reconstruction Accuracy: On the Animodel-Points benchmark, DualPM reduced the Root-Mean-Square Chamfer Distance for horses to 4.30 cm, down from 11.86 cm for 3D-Fauna.
  • Zero-Shot Generalization: A model trained only on horses performed surprisingly well on cows and sheep, outperforming models specifically trained on those categories (see Figure 6).
Figure 5. Comparison with the state-of-the-art. We compare our method with 3D-Fauna [33]. For visualization, we paint each point of the recovered point map P with its canonical coordinate. Our reconstructed shape aligns more closely with the object’s structure compared to 3D-Fauna and adapts well to large variations within the object category. Our method reliably recovers shapes outside its training distribution, such as foals and donkeys (rows 3 and 4). It also performs well on very challenging poses (rows 1 and 2), where 3D-Fauna completely fails.

Limitations and Future Work

  • Canonical Alignment: Currently, if multiple synthetic models are used for training, they must all share the same canonical space, which requires manual alignment.
  • External Occlusions: The model is designed for self-occlusion (amodal reconstruction) but is not yet specifically trained to handle occlusions by other objects (e.g., a horse standing behind a fence).
  • Expectation Bias: In highly ambiguous cases for invisible regions, the model predicts the "average" or expected shape, which can occasionally lead to overly smooth or unrealistic geometry for the hidden parts.

Summary updated

Abstract

The choice of data representation is a key factor in the success of deep learning in geometric tasks. For instance, DUSt3R recently introduced the concept of viewpoint-invariant point maps, generalizing depth prediction and showing that all key problems in the 3D reconstruction of static scenes can be reduced to predicting such point maps. In this paper, we develop an analogous concept for a very different problem: the reconstruction of the 3D shape and pose of deformable objects. To this end, we introduce Dual Point Maps (DualPM), where a pair of point maps is extracted from the same image-one associating pixels to their 3D locations on the object and the other to a canonical version of the object in its rest pose. We also extend point maps to amodal reconstruction to recover the complete shape of the object, even through self-occlusions. We show that 3D reconstruction and 3D pose estimation can be reduced to the prediction of DualPMs. Empirically, we demonstrate that this representation is a suitable target for deep networks to predict. Specifically, we focus on modeling quadrupeds, showing that DualPMs can be trained purely on synthetic 3D data, consisting of one or two models per category, while generalizing effectively to real images. With this approach, we achieve significant improvements over previous methods for the 3D analysis and reconstruction of such objects.
BibTeX
@inproceedings{kaye2025dualpm,
  author = {Ben Kaye and Tomas Jakab and Shangzhe Wu and Christian Rupprecht and Andrea Vedaldi},
  title = {{DualPM: Dual Posed-Canonical Point Maps for 3D Shape and Pose Reconstruction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00602},
}
Figure