Rethinking the Data Annotation Process for Multiview 3D Pose Estimation with Active Learning and Self-Training

Qi Feng, Kun He, He Wen, Cem Keskin, Yuting YeWACV 2023 · OpenReview

Abstract

Pose estimation of the human body and hands is a fundamental problem in computer vision, yet learning-based solutions are often constrained by the massive requirement for annotated data. This work presents a framework to improve the efficiency of the data annotation process for 3D pose estimation using Active Learning (AL) in multiview settings. By leveraging multiview geometry, we propose novel AL strategies that identify the most informative frames for human labeling. Furthermore, we incorporate a self-training (ST) loop that utilizes high-confidence pseudo-labels to augment the training set. Our approach demonstrates significant performance gains on the CMU Panoptic Studio and InterHand2.6M benchmarks, reducing annotation costs by up to 80% and turn-around time by 60% while maintaining state-of-the-art accuracy.

Introduction

The advancement of deep learning has significantly improved the accuracy of 3D human pose estimation. However, these models require vast amounts of high-quality labeled data. In multiview environments, where dozens of cameras capture a single scene, the annotation process is particularly grueling. A human annotator may spend a full minute labeling just one frame of 3D motion across multiple views.

The core challenge is efficiency: not all captured frames are equally informative. Redundant frames, such as a subject standing still, contribute little to model improvement. This paper addresses the bottleneck by introducing an Active Learning framework tailored for the geometric complexities of 3D pose estimation.

The Proposed Framework

The system architecture is designed to minimize human intervention by intelligently partitioning the data pool into frames requiring human expertise and those that can be handled automatically.

The proposed framework integrates Active Learning (AL) and Self-Training (ST) to optimize the annotation loop.

As shown in the figure above, the framework begins with an unlabeled pool of gathered raw data. A trained model performs inference to generate initial pseudo-labels. The Active Learning (AL) and Self-Training (ST) module then evaluates these labels. "Bad" or uncertain pseudo-labels are sent for human annotation, while "good" or high-confidence pseudo-labels are used for data augmentation. Both sets of labels eventually populate the labeled pool used to retrain the pose estimation model in an iterative loop.

Active Learning and Self-Training Strategies

The authors propose two primary AL strategies that utilize multiview geometry:

  1. CoreSet-Poses (Diversity-based): This strategy ensures the labeled dataset covers a wide range of the "pose space." By calculating geometric distances between 3D poses, the system selects a diverse set of actions (e.g., sitting, jumping, reaching) to avoid redundant labeling.
  2. Multi-View Consistency (OURS-MC): This uncertainty-based approach uses triangulation error as a proxy for model confidence. If 2D predictions from different cameras do not converge to a single 3D point, the model is deemed uncertain, and the frame is flagged for human annotation.

To further boost efficiency, the Self-Training (ST) component identifies frames where the model is exceptionally confident (low triangulation error). These frames receive automated "pseudo-labels," which are added to the training set without human cost. To prevent "model drift," a strict inlier requirement ensures that only highly consistent multiview predictions are used.

Experimental Results

The framework was evaluated on the CMU Panoptic Studio (body pose) and InterHand2.6M (hand pose) datasets. The results indicate that the proposed multiview strategies significantly outperform standard single-view AL techniques, such as Maximum Peak Entropy (MPE).

Comparison of the proposed AL-based annotation process against conventional annotation on the CMU Panoptic dataset.

As illustrated in the performance comparison above, the AL-based annotation process reaches a 3D Mean Keypoint Prediction Error (MKPE) of approximately 2.7mm in just 20 hours, whereas the conventional annotation process requires 50 hours to achieve the same level of precision. This represents a 60% reduction in turn-around time. Furthermore, the system achieved an 80% reduction in total annotation cost on the CMU Panoptic benchmark.

Conclusion

By rethinking the annotation process through the lens of multiview geometry, this work demonstrates that significant efficiency gains are possible in 3D pose estimation. The combination of uncertainty-based Active Learning and high-confidence Self-Training allows for the creation of high-performance models with a fraction of the human effort typically required. Future work may involve incorporating temporal information from video sequences to further reduce data redundancy and exploring applications in uncalibrated, "in-the-wild" camera setups.

Summary updated

Abstract

Pose estimation of the human body and hands is a fundamental problem in computer vision, and learning-based solutions require a large amount of annotated data. In this work, we improve the efficiency of the data annotation process for 3D pose estimation problems with Active Learning (AL) in a multiview setting. AL selects examples with the highest value to annotate under limited annotation budgets (time and cost), but choosing the selection strategy is often nontrivial. We present a framework to efficiently extend existing single-view AL strategies. We then propose two novel AL strategies that make full use of multiview geometry. Moreover, we demonstrate additional performance gains by incorporating pseudo-labels computed during the AL process, which is a form of self-training. Our system significantly outperforms simulated annotation baselines in 3D body and hand pose estimation on two large-scale benchmarks: CMU Panoptic Studio and InterHand2.6M. Notably, on CMU Panoptic Studio, we are able to reduce the turn-around time by 60% and annotation cost by 80% when compared to the conventional annotation process.
BibTeX
@inproceedings{feng2023rethinking,
  author = {Qi Feng and Kun He and He Wen and Cem Keskin and Yuting Ye},
  title = {{Rethinking the Data Annotation Process for Multiview 3D Pose Estimation with Active Learning and Self-Training}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=_6kxt_vxNO},
}
Figure