TL;DR
DualPM introduces a novel representation for 3D reconstruction that maps image pixels to two distinct 3D points: one in the observed camera space and one in a neutral "canonical" pose. By predicting these "Dual Point Maps," the model can simultaneously recover the full 3D shape and articulated pose of deformable objects—even hidden parts—while training exclusively on minimal synthetic data.
What Problem Does This Paper Address?
Reconstructing the 3D shape and pose of deformable objects (like animals) from a single 2D image is a classic "ill-posed" problem in computer vision. While recent breakthroughs like DUSt3R have simplified static scene reconstruction by predicting "point maps" (mapping pixels to 3D coordinates), deformable objects add a layer of complexity: articulation.
Traditional methods often try to regress bone rotations for a skeleton or fit a category-level template. However, these approaches struggle with the non-linear nature of 3D rotations and often fail to generalize to unusual poses or shapes not seen during training. Furthermore, standard point maps only capture the visible surface, leaving the "back" of the object a mystery.
Methods
The authors propose Dual Point Maps (DualPM). Instead of predicting one 3D coordinate per pixel, the network predicts two:
- Posed Point Map ($P$): The 3D location of the point in the camera's reference frame (the "where is it now" map).
- Canonical Point Map ($Q$): The 3D location of that same point on a "neutral" version of the object (the "where would it be if it were standing still" map).
As shown in Figure 1, the difference between these two maps ($P - Q$) effectively defines the deformation field, making pose estimation a simple subtraction problem.
To handle the "missing" parts of the object, the authors introduce Amodal Point Maps. Inspired by "depth peeling" in computer graphics, the model predicts multiple layers of points for each pixel. If a camera ray passes through a horse's front leg and then its torso, the model predicts coordinates for both intersection points. This is managed via a layered representation and associated opacity maps (Figure 2).
The architecture uses a pre-trained feature extractor (combining DINOv2 and Stable Diffusion features) to first predict the canonical map $Q$, and then uses $Q$ as a feature to predict the posed map $P$. This conditioning helps the model generalize better to real-world images.
Datasets
- Training: The models are trained purely on synthetic data generated from just one or two rigged 3D models per category (e.g., one horse model) from the Animodel dataset.
- Evaluation:
- PASCAL VOC: Used for real-world qualitative results and quantitative keypoint transfer tasks.
- Animodel-Points: A new, more robust benchmark introduced by the authors to evaluate point cloud reconstruction accuracy using bi-directional Chamfer Distance.
Key Insights and Ideas
- Pixel-Labeling is Easier: The authors argue that predicting pixel-aligned 3D coordinates is a much simpler task for a neural network than regressing a chain of abstract 3D bone rotations. It turns a complex global optimization problem into a dense prediction task.
- Canonical Invariance: The canonical map $Q$ is viewpoint- and pose-invariant. This makes it a very stable intermediate representation that simplifies the subsequent prediction of the posed map $P$.
- Synthetic-to-Real Generalization: By focusing on geometric point maps rather than raw pixels, the model generalizes remarkably well from a single synthetic 3D asset to diverse real-world photos of different breeds and ages (e.g., training on an adult horse but succeeding on a foal).
Key Results
DualPM significantly outperforms existing state-of-the-art methods like 3D-Fauna across multiple metrics:
- Keypoint Transfer: On PASCAL VOC, DualPM achieved a 73.2% [email protected] for horses, compared to 53.9% for 3D-Fauna.
- Reconstruction Accuracy: On the Animodel-Points benchmark, DualPM reduced the Root-Mean-Square Chamfer Distance for horses to 4.30 cm, down from 11.86 cm for 3D-Fauna.
- Zero-Shot Generalization: A model trained only on horses performed surprisingly well on cows and sheep, outperforming models specifically trained on those categories (see Figure 6).
Limitations and Future Work
- Canonical Alignment: Currently, if multiple synthetic models are used for training, they must all share the same canonical space, which requires manual alignment.
- External Occlusions: The model is designed for self-occlusion (amodal reconstruction) but is not yet specifically trained to handle occlusions by other objects (e.g., a horse standing behind a fence).
- Expectation Bias: In highly ambiguous cases for invisible regions, the model predicts the "average" or expected shape, which can occasionally lead to overly smooth or unrealistic geometry for the hidden parts.
Summary updated