TL;DR
This paper introduces a novel Active Transfer Learning (ATL) framework to efficiently adapt human pose estimation models to individual video domains with minimal annotation effort. By combining temporal heatmap continuity and whole-body pose unnaturalness criteria for sample selection, and introducing a dynamic weighting strategy, the method significantly improves learning efficiency and accuracy compared to existing approaches.
What Problem Does This Paper Address?
Human Pose (HP) estimation is crucial for many applications, but pre-trained models often struggle with new video domains due to a "domain gap." Manually annotating every frame in a video for HP estimation is prohibitively time-consuming and expensive. While Transfer Learning (TL) helps adapt models, and Active Learning (AL) reduces annotation costs, efficiently combining them for video-specific HP estimation remains a challenge. Existing AL methods might select redundant samples from videos, and current TL approaches may not be optimized for the continuous nature of video data. This paper aims to bridge this gap by developing an efficient ATL framework tailored for video-specific HP estimation.
Methods
The core of this work is an Active Transfer Learning (ATL) cycle designed for video-specific adaptation. The framework iteratively performs the following steps:
- HP Estimation: A pre-trained HP estimator is used to infer poses on unlabeled video frames.
- Active Selection: This is where the novelty lies. Instead of random sampling or relying solely on existing uncertainty/representativeness criteria, the authors propose three new criteria:
- Temporal Heatmap Continuity (THC): Measures uncertainty based on how much the estimated heatmaps for keypoints change across consecutive frames. Inconsistent heatmaps suggest uncertainty.
- Whole-body Pose Unnaturalness (WPU): Quantifies pose-level uncertainty by training an AutoEncoder (AE) on ground-truth poses. Unnatural poses result in higher reconstruction errors from the AE, indicating uncertainty. This method is noted for its computational efficiency compared to prior work.
- Dynamic Uncertainty Weighting (DUW): This criterion integrates the proposed THC and WPU with the representativeness-based Core-Set sampling [53]. It dynamically weights the contribution of uncertainty and diversity based on the current performance of the HP estimator (represented by $G_c$). This allows for prioritizing coverage in early stages and focusing on uncertain samples later.
- Human Annotation: Selected samples are manually annotated.
- ACFT-based Retraining: The HP estimator is retrained using both newly annotated and previously labeled samples. The authors refine the ACFT [66] approach by proposing a novel stopping criterion (SC) and a dynamic epoch calculation for retraining.
- Stopping Criterion (SC): A new SC, $SC_{All}$, is introduced to determine when to terminate the ATL cycle. It aims to stop when the HP estimation accuracy (measured by OKS) reaches a user-defined threshold across all samples (labeled and unlabeled), ensuring the desired accuracy is met.
Datasets
The proposed method was evaluated on two large-scale multi-person 2D HP estimation datasets:
- PoseTrack21 [17]: Used for training and validation. The experiments utilized 579 training videos and 170 validation videos, with approximately 30 annotated frames per video for evaluation.
- JRDB-Pose [60]: Also used for training and evaluation. The dataset was split into 10 training, 2 validation, and 15 testing videos, with the first 150 frames per video used. Poses in JRDB-Pose consist of 17 keypoints.
The paper does not introduce any new datasets.
Key Insights and Ideas
- Temporal Dynamics are Key for Video HP: The paper highlights that simply treating video frames as independent images misses crucial temporal information. Incorporating temporal consistency in heatmaps (THC) and pose structure (WPU) leads to more robust uncertainty estimation.
- Unnaturalness as a Proxy for Uncertainty: Using an AutoEncoder trained on ground-truth poses to detect "unnatural" poses is an elegant and computationally efficient way to quantify pose-level uncertainty, especially for detecting subtle errors that might be missed by keypoint-specific uncertainty measures.
- Adaptive Sampling for Efficiency: The dynamic weighting of uncertainty and diversity (DUW) is crucial. It allows the system to first cover the general pose distribution in a video and then focus on the most challenging or informative samples as the model improves, leading to faster convergence.
- Tailored Stopping Criteria for Practicality: A fixed stopping criterion can be too early or too late. The proposed $SC_{All}$ criterion, which aims for a user-defined accuracy threshold across all samples, provides a more practical way to ensure the desired performance level is achieved before halting the annotation and retraining process.
Key Results
- The proposed method ("THC+WPU+DUW") significantly outperforms existing ATL methods, including random sampling, uncertainty-based methods (LC, MPE, TPC), and representativeness-based methods (k-means, Core-Set), on both PoseTrack21 and JRDB-Pose datasets.
- On PoseTrack21, the method achieved an [email protected] of approximately 98% with only 20% labeled samples, demonstrating high learning efficiency.
- Ablation studies confirmed that each component (THC, WPU, DUW) contributes to the overall performance, with the combination yielding the best results.
- The dynamic weighting in DUW was shown to be superior to a fixed weighting strategy.
- The proposed $SC_{All}$ criterion effectively terminates ATL when a high AP is reached, ensuring user-defined accuracy, unlike the Min-error criterion which can stop prematurely.
Limitations and Future Work
- Limitations:
- The paper simulates human annotation by using ground truth. In a real-world scenario, human annotator variability and errors could introduce additional challenges.
- The method relies on a pre-trained HP estimator. The performance of the final adapted model is still dependent on the quality of the initial pre-trained model.
- Future Work:
- Integrate video-based HP estimation methods to further enhance performance.
- Explore semi-supervised learning techniques to further reduce annotation costs.
- Investigate more sophisticated strategies for combining THC and WPU, as suggested by the ablation study showing different performance characteristics for "increase," "const," and "decrease" scenarios.
Summary updated