Abstract
Reconstructing 3D animals from single images is a significant challenge due to the lack of large-scale 3D scan datasets. Current methods often rely on the SMAL parametric model, which is limited by its origin in toy figurines and struggles to capture the diverse shape variations of real-world breeds. We present a novel coarse-to-fine framework that combines the stability of parametric models with the flexibility of non-parametric refinement. By employing a hierarchical Graph Convolutional Network (GCN) that leverages local image features, our system achieves state-of-the-art performance in dog reconstruction using only weak 2D supervision.
Introduction
3D animal reconstruction is inherently more difficult than human reconstruction because of the scarcity of 3D ground truth data. Most existing approaches utilize SMAL, a parametric model built from 3D scans of just 41 toy figurines. This leads to a "toy problem": real-world animals, particularly diverse dog breeds, exhibit shape variations that the rigid SMAL model cannot fully accommodate. Consequently, even optimal SMAL parameters often result in meshes that misalign with 2D keypoints or silhouettes. This research introduces a method to break free from these constraints to capture fine-grained anatomical details.
Methods
The proposed framework utilizes a two-stage approach to balance parametric stability with non-parametric detail.
Coarse Estimation Stage
The network first regresses global SMAL parameters (pose, shape, translation) and camera parameters from a global image feature. To improve initial silhouette alignment, the authors implement the Tversky loss. This loss function effectively manages the imbalance between foreground and background pixels, preventing the common issue of "over-fattening" the estimated mesh during the initial fit.
Mesh Refinement Stage
The coarse mesh is then processed by a Mesh Refinement GCN (MRGCN). Unlike standard GCNs, this model incorporates a local feature extractor. It projects each 3D vertex back onto the 2D image feature map to sample specific visual data at that precise location.
To overcome the limited receptive field of traditional GCNs, the authors designed a hierarchical encoder-decoder structure for the mesh. By performing downsampling and upsampling on the graph, the network can effectively propagate information across the entire body, capturing both global structure and local refinements. The entire system is trained using weak supervision, requiring only 2D keypoints and silhouettes.
Key Results
The model was evaluated on the StanfordExtra, Animal Pose, and BADJA datasets. It demonstrated significant improvements over existing baselines:
- State-of-the-Art Accuracy: On StanfordExtra, the model achieved an Intersection-over-Union (IOU) of 81.6, compared to the previous best of 74.2 (WLDO).
- Pose Precision: The [email protected] (Percentage of Correct Keypoints) reached 83.4, surpassing the 78.8 achieved by prior methods.
- Computational Efficiency: The regression-based refinement is remarkably fast, processing the entire test set in approximately 64.5 seconds, whereas traditional test-time optimization methods can take over 10,000 seconds to reach lower accuracy levels.
Discussion and Conclusion
The integration of local features and a hierarchical GCN allows the model to adjust the mesh to match specific fur contours and limb positions that global features often miss. While the model sets a new benchmark, challenges remain in "extreme" scenarios, such as severe occlusions or views directly from the animal's rear. Future research may explore modeling 3D ambiguity or incorporating temporal data from video sequences to enhance realism and robustness in dynamic environments.
Summary updated