Abstract
Reconstructing the 3D pose and shape of animals from a single "in-the-wild" image remains a significant challenge in computer vision due to extreme morphological variety and a lack of large-scale 3D ground truth data. This paper presents an end-to-end method for 3D dog reconstruction that introduces a new parametric model, SMBLD (Skinned Multi-Breed Linear Model for Dogs), and a novel training loop utilizing Expectation Maximization (EM). By learning a multi-modal 3D shape prior directly from 2D data, the system achieves state-of-the-art results across diverse dog breeds with fast, feed-forward inference.
Introduction
While 3D human body reconstruction has advanced rapidly through datasets like SMPL, animal reconstruction faces unique hurdles. Dogs, for instance, exhibit vast differences in proportions—from Chihuahuas to Great Danes—despite sharing a similar skeletal structure. Furthermore, high-quality 3D scans of various breeds are scarce. Traditional methods often rely on slow, per-image optimization that struggles with the low-quality, occluded images typical of internet data. This work proposes a fast, robust alternative that generalizes across the "in-the-wild" diversity of canine morphology.
Methodology: SMBLD and EM-in-the-Loop
The core of the proposed pipeline is the SMBLD (Skinned Multi-Breed Linear Model for Dogs). Extending the SMAL model, SMBLD incorporates explicit scaling parameters for subparts (e.g., limb and tail length), allowing the model to capture breed-specific traits more accurately than PCA-based shape parameters alone.
To overcome the lack of 3D training data, the authors introduce an EM-in-the-loop training strategy. The network predicts shape parameters which are iteratively clustered into a Multi-modal Mixture of Gaussians (MoG) prior. This allows the model to learn "typical" shapes for different breed clusters directly from 2D annotations. A differentiable renderer is used to project the 3D model back into 2D, minimizing losses against ground-truth silhouettes and joints.
Experimental Results
The model was evaluated on StanfordExtra, a new large-scale benchmark containing over 20,000 images with manual 2D joint and silhouette annotations. The method achieved a state-of-the-art Intersection over Union (IoU) of 74.2% and a Percentage of Correct Keypoints (PCK) of 78.8%.
Beyond accuracy, the feed-forward nature of the network makes it orders of magnitude faster than previous optimization-based approaches. It also demonstrated strong generalization on the Animal Pose dataset without additional fine-tuning, proving the robustness of the learned multi-modal priors.
Discussion and Limitations
Despite its success, the model has limitations. The current SMBLD skeletal structure lacks sufficient degrees of freedom to accurately model "floppy" ears. Additionally, performance can degrade in complex scenes with heavy motion blur or multiple overlapping animals. Future work may involve extending the EM formulation to video sequences to leverage temporal and multi-view constraints.
Conclusion
By combining a flexible parametric model with a novel EM-based learning strategy, this research demonstrates that detailed 3D animal priors can be learned effectively from 2D data. This approach paves the way for scalable 3D reconstruction of diverse animal species without the need for exhaustive 3D scanning.
Summary updated