Decompose to Generalize: Species-Generalized Animal Pose Estimation

Guangrui Li, Yifan Sun, Zongxin Yang, Yi YangICLR 2023 · OpenReview

TL;DR

The Decompose-to-Generalize (D-Gen) framework improves animal pose estimation for unseen species by grouping body joints into "concepts." By promoting interaction between related joints (like eyes and nose) and suppressing distraction from inconsistent ones (like varying leg lengths), the model generalizes significantly better to novel animals than standard domain generalization methods.

The Challenge of Cross-Species Generalization

Animal pose estimation—identifying the location of joints like knees, eyes, and tails—is vital for biology and ecology. However, the animal kingdom contains millions of species, making it impossible to collect and annotate data for every single one.

The core challenge is cross-species generalization: training a model on a few species (e.g., dogs and cats) and having it work on a completely different one (e.g., a tiger or a cow). Standard "Domain Generalization" (DG) techniques usually focus on making models ignore "style" (like fur color or background). The authors of this paper argue that for animals, the real problem lies in the relation between joints.

Proportional length of body parts across different animal families.

Some relations are consistent across species (e.g., eyes are always near the nose), providing helpful "mutual confirmation." Others are highly inconsistent. For example, the distance between a hip and a knee varies wildly between a giraffe and a pig. These inconsistent relations act as "distractions" that confuse the model when it encounters a new body shape.

The Decompose-to-Generalize (D-Gen) Method

To address this, the authors propose D-Gen, a two-stage approach designed to embrace consistent relations while isolating inconsistent ones.

Grouping animal joints into distinct anatomical concepts.
  1. Joint Decomposition: The model divides the body's joints into several "joint concepts." The paper explores three ways to do this:
  2. Heuristic: Based on human anatomical knowledge (e.g., grouping "head," "forelegs," and "hind legs").
  3. Geometric: Using spectral clustering based on the physical distance between joints in the training data.
  4. Attention-based (The Winner): A novel "pixel-to-concept" attention module that learns which joints naturally belong together by looking at feature affinities.
  5. Network Split: Instead of one giant network head predicting all joints, D-Gen splits the top layers of the backbone into multiple branches. Each branch is responsible for one "concept."
    • Intra-concept interaction: Joints within the same branch can "talk" to each other, helping the model confirm their locations.
  6. Inter-concept suppression: Because joints from different concepts are in different branches, they cannot distract or interfere with each other during the optimization process.

Key Insights and Ideas

  • The "Distraction" Hypothesis: The authors used gradient analysis to prove that joints from different concepts often have "conflicting gradients" during training. By splitting the network into branches, they effectively reduced this conflict, allowing the model to converge on more robust features.
  • Attention Beats Intuition: Surprisingly, the attention-based decomposition outperformed human-defined anatomical groups. The model discovered that grouping the "left eye" with "easy-to-distinguish" joints in one concept and the "right eye" in another actually helped it discriminate between the two better than putting both eyes in a "head" group.
  • Beyond Style: Most DG research focuses on image appearance (texture/color). This paper shifts the focus to structural consistency, proving that how a model understands the "geometry" of an object is just as important for generalization as how it understands "pixels."

Datasets and Key Results

The researchers evaluated D-Gen on three major benchmarks: AP-10K (54 species), the Animal Pose Dataset (5 species), and Animal Kingdom (highly diverse across classes).

D-Gen consistently outperformed both standard baselines (ERM) and state-of-the-art domain generalization methods (like MixStyle and SWAD).

  • AP-10K (Intra-family): D-Gen achieved an Average Precision (AP) of 55.0, compared to 52.4 for the standard baseline.
  • Animal Pose Dataset: D-Gen reached 28.9 AP, a significant jump from the 23.4 AP of the baseline.
  • Few-shot Power: In one test on the "Bobcat" species (with only 150 images), D-Gen actually outperformed the "Oracle" model (a model trained specifically on that species), showing that the learned structural concepts are incredibly powerful when data is scarce.

Limitations and Future Work

While highly effective, D-Gen has limitations. The model still struggles in wild environments where animals are heavily hidden by vegetation or terrain (Extreme Occlusion). Furthermore, the current method assumes a "well-aligned joint space." It works well across mammals because they share similar skeletons, but generalizing from a four-legged mammal to a bird or a fish remains a significant hurdle for future research. Finally, splitting the network into multiple branches slightly increases computational overhead, though the authors note this increase is modest.

Summary updated

Abstract

This paper challenges the cross-species generalization problem for animal pose estimation, aiming to learn a pose estimator that can be well generalized to novel species. We find the relation between different joints is important with two-fold impact: 1) on the one hand, some relation is consistent across all the species and may help two joints mutually confirm each other, e.g., the eyes help confirm the nose and vice versa because they are close in all species. 2) on the other hand, some relation is inconsistent for different species due to the species variation and may bring severe distraction rather than benefit. With these two insights, we propose a Decompose-to-Generalize (D-Gen) pose estimation method to break the inconsistent relations while preserving the consistent ones. Specifically, D-Gen first decomposes the body joints into several joint concepts so that each concept contains multiple closely-related joints. Given these joint concepts, D-Gen 1) promotes the interaction between intra-concept joints to enhance their reliable mutual confirmation, and 2) suppresses the interaction between inter-concept joints to prohibit their mutual distraction. Importantly, we explore various decomposition approaches, i.e., heuristic, geometric and attention-based approaches. Experimental results show that all these decomposition manners yield reasonable joint concepts and substantially improve cross-species generalization (and the attention-based approach is the best).
BibTeX
@inproceedings{li2023decompose,
  author = {Guangrui Li and Yifan Sun and Zongxin Yang and Yi Yang},
  title = {{Decompose to Generalize: Species-Generalized Animal Pose Estimation}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=nQai_B1Zrt},
}
Figure