Abstract
Annotating human poses in realistic scenes is very time consuming, yet necessary for training human pose estimators. We propose to address this problem in an active learning framework, which alternates between requesting the most useful annotations among a large set of unlabelled images, and re-training the pose estimator. To this end, (1) we propose an uncertainty estimator specific for body joint predictions, which takes into account the spatial distribution of the responses of the current pose estimator on the unlabelled images; (2) we propose a dynamic combination of influence and uncertainty cues, where their weights vary during the active learning process according to the reliability of the current pose estimator; (3) we introduce a computer assisted annotation interface, which reduces the time necessary for a human annotator to click on a joint by discretizing the image into regions generated by the current pose estimator. Experiments using the MPII and LSP datasets with both simulated and real annotators show that (1) the proposed active selection scheme outperforms several baselines; (2) our computer-assisted interface can further reduce annotation effort; and (3) our technique can further improve the performance of a pose estimator even when starting from an already strong one.
Introduction
Training state-of-the-art human pose estimators, such as Convolutional Pose Machines (CPM), requires massive amounts of labeled data. However, annotating poses is a labor-intensive task, requiring a human to precisely click on 14–16 individual body joints per person. At an average rate of one minute per pose, scaling datasets to hundreds of thousands of images is prohibitively expensive.
While Active Learning (AL)—the process of having a model "ask" for labels on only the most informative images—has been successful in image classification and object detection, its application to the high-dimensional, spatial task of pose estimation is relatively unexplored. This paper introduces the first active learning framework specifically designed for human pose estimation, aiming to achieve high performance while significantly reducing annotation costs.
Proposed Framework
The proposed framework alternates between selecting the most informative unlabelled images for annotation and re-training the pose estimator. The system architecture is divided into three main components: a pose estimator, an active selection strategy, and a computer-assisted annotation interface.
1. Multiple Peak Entropy (MPE)
Standard uncertainty measures, such as the maximum probability of a single pixel, are often inadequate for pose estimation. Heatmaps generated by models can be diffuse or multi-modal, reflecting confusion between similar joints (e.g., left vs. right knee).
The proposed Multiple Peak Entropy (MPE) metric identifies all local maxima (peaks) in the predicted heatmap. It then calculates entropy across these peaks, effectively targeting images where the model is uncertain between multiple distinct spatial locations.
2. Dynamic Combination of Cues
The selection strategy balances two cues:
- Influence: This selects "prototypical" images that are representative of the unlabelled pool.
- Uncertainty (MPE): This selects "hard" images where the model's current predictions are most confused.
A key innovation is the dynamic weighting of these cues. Early in training, when the model is weak, the system favors Influence to build a foundational understanding of human poses. As the model matures and becomes more reliable, the weight shifts toward Uncertainty to refine performance on difficult cases.
3. Computer-Assisted Annotation Interface
To further reduce manual effort, the authors introduce a Voronoi Diagram (VD)-based interface. Instead of requiring a pixel-perfect click, the interface discretizes the image into regions based on the model's predicted peaks. The annotator only needs to click within the correct region. If the model's candidates are incorrect, the user can still provide a manual correction.
Experimental Results
The researchers evaluated their approach on the MPII Human Pose and Extended Leeds Sports Pose (LSP) datasets.
- Efficiency: The system achieved 80% of the full-dataset accuracy using only 23% of the total annotation time.
- Annotation Speedup: The Voronoi interface reduced the time to label a single joint by approximately 33% without compromising accuracy.
- Selection Performance: The MPE metric consistently outperformed standard active learning baselines, such as "Highest Probability" or "Best vs. Second Best," across both benchmarks.
- Real-World Robustness: Experiments with real human annotators confirmed the method's robustness to labeling noise.
Conclusion
This work demonstrates that active learning, combined with specialized uncertainty metrics and intelligent UI design, can dramatically lower the barriers to training high-performance pose estimators. By focusing on spatially-aware uncertainty (MPE) and evolving selection strategies, the framework slashes annotation costs while maintaining state-of-the-art potential. Future work may extend these principles to multi-person pose estimation and more complex joint association tasks.
Summary updated