Coarse-to-fine Animal Pose and Shape Estimation

Chen Li, Gim Hee Lee

Estimating the 3D pose and shape of animals from a single 2D image is a notoriously difficult task. While the computer vision community has amassed massive datasets of 3D human scans to train robust models, collecting similar data for animals is nearly impossible—you cannot easily ask a tiger or a stray dog to stand still in a 3D scanner.

To bridge this gap, researchers from the National University of Singapore have developed a novel coarse-to-fine framework for reconstructing 3D animal meshes using only 2D supervision. By combining a parametric model with a hierarchical Graph Convolutional Network (GCN) that utilizes both global and local image features, their system captures fine-grained anatomical details that traditional models often miss.

The Challenge: The "Representation Gap"

Because real 3D animal scans are scarce, researchers typically rely on SMAL, a parametric model learned from 3D scans of toy animals. However, SMAL suffers from a significant "representation gap." Because it was built from a limited number of toys, it struggles to represent the vast diversity of real-world animal shapes, proportions, and breeds.

When applied to real images, this limitation often leads to "poor fittings." The estimated 3D mesh might look generally like a dog, but it won't accurately align with the specific animal's silhouette or keypoints in the 2D image.

Methodology: A Coarse-to-Fine Approach

To overcome the limitations of toy-based parametric models, the researchers propose a two-stage approach that blends the stability of parametric models with the flexibility of non-parametric refinement.

Overview of the coarse-to-fine framework, illustrating the initial SMAL parameter estimation followed by the Mesh Refinement GCN (MRGCN).

1. Coarse Estimation Stage

The network first uses a ResNet-50 encoder to extract global features from the input image. It then regresses the pose, shape, and translation parameters of the SMAL model. This provides a "good enough" starting mesh that captures the general posture and orientation of the animal, preventing the subsequent steps from degrading into an unrecognizable "vertex soup."

2. Fine Refinement Stage (MRGCN)

To fix the anatomical details that SMAL misses, the authors introduce a Mesh Refinement Graph Convolutional Network (MRGCN). This stage features two major innovations:

  • Encoder-Decoder Structure: Standard GCNs have a limited "receptive field," meaning they only look at immediate neighboring vertices. The MRGCN uses downsampling and upsampling to capture hierarchical features across the entire mesh, allowing for more coherent structural deformations.
  • Local Feature Extractor: A single global vector representing an entire image cannot tell a specific vertex on a dog's ear how to move. To solve this, the model projects the coarse 3D vertices back onto the 2D image to "grab" local, pixel-level features. This allows the network to see exactly where a leg or ear is misaligned and deform the mesh accordingly.

Weak Supervision & Tversky Loss

Because there is no 3D ground truth available for training, the model relies entirely on 2D keypoints and silhouettes. Standard silhouette losses (like $L_1$ or $L_2$) often result in a "fat mesh" bias, where the network over-estimates the animal's size to avoid the heavy penalty of missing foreground pixels. The authors mitigate this by employing the Tversky loss, which allows them to finely tune the balance between false positives and false negatives.

Results and Performance

The model was trained and evaluated on StanfordExtra (a large-scale dog dataset covering 120 breeds), as well as the Animal Pose and BADJA datasets to test generalization across different animal types and video sequences.

Qualitative comparison showing the initial coarse mesh estimation versus the final refined mesh, demonstrating improved alignment with the animal's true shape.

The approach achieved state-of-the-art (SOTA) results across all tested datasets. On the StanfordExtra dataset, the method yielded:

  • IOU (Intersection over Union): Improved to 81.6, significantly outperforming the previous SOTA (WLDO at 74.2).
  • PCK (Percentage of Correct Keypoints): Achieved an average score of 83.4, showing vastly superior alignment with the animal's actual joints compared to SMAL-only methods.

Qualitatively, the refinement stage visibly "tucks in" the mesh to match the animal's actual proportions. It correctly captures thinner legs, specific ear shapes, and distinct breed profiles that the base SMAL model simply could not represent.

Limitations and Future Work

While highly effective, the authors note a few remaining hurdles:

  • Extreme Occlusion: The model still struggles when significant parts of the animal are hidden or when the camera is positioned directly behind the subject.
  • 2D-to-3D Ambiguity: Because the system relies solely on 2D data, there can be multiple 3D shapes that fit a given 2D silhouette equally well.
  • Societal Impact: While this technology is highly useful for zoology, animation, or farming, the authors warn that inaccurate reconstructions could lead to misleading health monitoring if deployed in veterinary contexts without human oversight.

Future directions include modeling these inherent ambiguities by predicting multiple hypotheses or probability distributions over the model parameters, further pushing the boundaries of what is possible in 3D animal reconstruction.

Summary updated

Abstract

Most existing animal pose and shape estimation approaches reconstruct animal meshes with a parametric SMAL model. This is because the low-dimensional pose and shape parameters of the SMAL model makes it easier for deep networks to learn the high-dimensional animal meshes. However, the SMAL model is learned from scans of toy animals with limited pose and shape variations, and thus may not be able to represent highly varying real animals well. This may result in poor fittings of the estimated meshes to the 2D evidences, e.g. 2D keypoints or silhouettes. To mitigate this problem, we propose a coarse-to-fine approach to reconstruct 3D animal mesh from a single image. The coarse estimation stage first estimates the pose, shape and translation parameters of the SMAL model. The estimated meshes are then used as a starting point by a graph convolutional network (GCN) to predict a per-vertex deformation in the refinement stage. This combination of SMAL-based and vertex-based representations benefits from both parametric and non-parametric representations. We design our mesh refinement GCN (MRGCN) as an encoder-decoder structure with hierarchical feature representations to overcome the limited receptive field of traditional GCNs. Moreover, we observe that the global image feature used by existing animal mesh reconstruction works is unable to capture detailed shape information for mesh refinement. We thus introduce a local feature extractor to retrieve a vertex-level feature and use it together with the global feature as the input of the MRGCN. We test our approach on the StanfordExtra dataset and achieve state-of-the-art results. Furthermore, we test the generalization capacity of our approach on the Animal Pose and BADJA datasets. Our code is available at the project website.
BibTeX
@inproceedings{li2021coarse,
  author = {Chen Li and Gim Hee Lee},
  title = {{Coarse-to-fine Animal Pose and Shape Estimation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=DYUrabEuqM},
}
Figure