Pretraining boosts out-of-domain robustness for pose estimation

Alexander Mathis, Mert Yüksekgönül, Byron Rogers, Matthias Bethge, Mackenzie W. Mathis

Introduction

In computer vision, pose estimation—the task of identifying the location of specific body parts—has reached high levels of accuracy on standard benchmarks. However, these benchmarks often fail to reflect real-world laboratory or clinical settings. While datasets like COCO feature thousands of individuals, researchers in specialized fields often have many frames of just a few subjects. This leads to a "generalization gap": models perform exceptionally well on training individuals but fail when presented with new subjects of the same species. This paper investigates whether the standard practice of ImageNet pretraining can mitigate this issue.

Methodology and Datasets

The researchers utilized the DeepLabCut framework to evaluate several neural network architectures, including ResNets, MobileNetV2s, and EfficientNets. They compared two primary training strategies:

  1. Transfer Learning: Initializing models with weights pretrained on the ImageNet dataset.
  2. Training from Scratch: Initializing with random weights and training solely on the pose estimation task.

To rigorously test generalization, the authors introduced the Horse-10 dataset. It consists of 30 diverse Thoroughbred horses with 22 labeled body parts across 8,114 frames. The data is split such that 10 horses are used for training and "within-domain" testing, while 20 horses are held out entirely for "out-of-domain" (OOD) testing.

Figure 4: Evaluation framework and results. (A) The normalized error metric (nose-to-eye distance). (B, C) Error curves showing the impact of training data volume (5% vs 50%). (D) Qualitative pose estimation results on the Horse-10 dataset.

Performance was measured using a normalized error metric based on the distance between the horse's nose and eye, providing a scale-invariant assessment of accuracy.

The "Within-Domain" Illusion

A key finding of the study is that evaluating models only on "seen" individuals creates a false sense of security. When tested on the same individuals used during training, models trained from scratch perform nearly as well as those using transfer learning. This "within-domain" success masks a fundamental lack of robustness that only becomes apparent when the model encounters a new individual with different coat colors, lighting, or backgrounds.

Results: Pretraining as a Robustness Multiplier

The experimental results reveal that ImageNet pretraining is a powerful tool for OOD generalization. Pretrained models were found to be roughly twice as accurate on unseen horses compared to their "from scratch" counterparts.

Figure 1: Normalized Error vs. ImageNet Top-1 accuracy. The plot demonstrates a strong correlation between classification performance and pose estimation robustness. Note the significant gap between scratch-trained (solid top line) and pretrained (dashed top line) models on out-of-domain data.

Furthermore, the study found a strong linear correlation between a backbone's Top-1 accuracy on ImageNet and its OOD robustness in pose estimation. This suggests that the features learned for general object classification translate effectively into a deeper "understanding" of animal geometry.

Conclusion and Future Work

While ImageNet pretraining provides a massive boost to robustness, a significant "generalization gap" remains—errors on unseen individuals are still roughly three times higher than on seen ones. This highlights the ongoing challenge of creating truly universal pose estimation models. Future work may explore larger pretraining datasets, unsupervised domain adaptation, or task-specific pretraining to further close this gap. For practitioners, the message is clear: for real-world applications where subject diversity is expected, pretraining is not just a convenience—it is a necessity.

Summary updated

Abstract

Neural networks are highly effective tools for pose estimation. However, as in other computer vision tasks, robustness to out-of-domain data remains a challenge, especially for small training sets that are common for real-world applications. Here, we probe the generalization ability with three architecture classes (MobileNetV2s, ResNets, and EfficientNets) for pose estimation. We developed a dataset of 30 horses that allowed for both "within-domain" and "out-of-domain" (unseen horse) benchmarking - this is a crucial test for robustness that current human pose estimation benchmarks do not directly address. We show that better ImageNet-performing architectures perform better on both within- and out-of-domain data if they are first pretrained on ImageNet. We additionally show that better ImageNet models generalize better across animal species. Furthermore, we introduce Horse-C, a new benchmark for common corruptions for pose estimation, and confirm that pretraining increases performance in this domain shift context as well. Overall, our results demonstrate that transfer learning is beneficial for out-of-domain robustness.
BibTeX
@inproceedings{mathis2019pretraining,
  author = {Alexander Mathis and Mert Yüksekgönül and Byron Rogers and Matthias Bethge and Mackenzie W. Mathis},
  title = {{Pretraining boosts out-of-domain robustness for pose estimation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2019},
  doi = {10.1109/wacv48630.2021.00190},
}
Figure