Uncertainty Sampling for Action Recognition via Maximizing Expected Average Precision

Hanmo Wang, Xiaojun Chang, Lei Shi, Yi Yang, Yi-Dong ShenIJCAI 2018 · DOI

Abstract

Active learning for action recognition often relies on uncertainty sampling strategies that optimize for accuracy, despite the standard evaluation metric being Average Precision (AP). This paper introduces Uncertainty Sampling via maximizing expected Average Precision (USAP), a novel strategy that aligns the selection process with the evaluation metric. By treating unlabeled videos as Bernoulli random variables and employing dynamic programming to calculate expected AP, USAP significantly improves labeling efficiency and model performance on major benchmarks like HMDB51, UCF50, and Hollywood2.

Introduction

Labeling video data for action recognition is a resource-intensive task. Active Learning (AL) offers a solution by selecting only the most informative samples for manual annotation. However, a fundamental mismatch exists in current AL frameworks: while models are evaluated using Average Precision (AP), selection strategies typically optimize for Accuracy. Traditional uncertainty measures, such as Entropy, are mathematically tied to accuracy improvement, which may not translate to better ranking performance in retrieval-style tasks like action recognition.

Methodology: Aligning Selection with Evaluation

The authors propose Uncertainty Sampling via maximizing expected Average Precision (USAP) to bridge this gap. The method consists of three primary components:

  1. Probabilistic Labeling: Unlabeled videos are modeled as independent Bernoulli random variables. The model's current confidence score for a class is treated as the probability of the "true" label being positive.
  2. Expected AP Calculation: Unlike instance-level accuracy, AP is a rank-based metric. The authors derive a formula to calculate the expected value of AP over the joint distribution of these probabilistic labels.
  3. Efficient Optimization: To overcome the exponential complexity of calculating expected AP, a dynamic programming (DP) algorithm is introduced, reducing complexity to $O(cn^3)$. Furthermore, a "Fast Screening Rule" based on precision@n is used to filter the candidate pool before applying the DP calculation.

Experimental Evaluation

The USAP method was evaluated against several baselines, including Random sampling (Rand), Maximum Posterior probability (Pmax), and Entropy-based sampling (Entro), across three standard benchmarks: HMDB51, UCF50, and Hollywood2.

Performance comparison on HMDB51 and UCF50 datasets across different labeling budgets.

As shown in the results for HMDB51 and UCF50, USAP consistently outperforms traditional uncertainty sampling methods as the labeling budget increases. The performance gap becomes more pronounced with more labeled samples, demonstrating superior labeling efficiency.

Performance comparison on the Hollywood2 dataset, highlighting USAP's robustness in noisy environments.

On the Hollywood2 dataset, which is known for its noise and multi-label complexity, USAP showed significant advantages. For instance, USAP reached a 58.11% mAP with only 24c labeled videos, whereas traditional methods like Pmax and Entro required nearly 28c to 30c labels to approach similar performance levels.

Discussion and Key Insights

The core insight of this work is that "uncertainty" is relative to the evaluation metric. By aligning the selection criteria with the rank-based nature of AP, USAP avoids the pitfalls of accuracy-centric sampling. This is particularly evident in noisy datasets like Hollywood2, where standard AL methods can sometimes perform worse than random sampling due to outliers. USAP's focus on expected ranking makes it more robust.

While USAP is more computationally intensive than simple entropy measures, the introduction of the Fast Screening Rule makes it feasible for practical use. Future work may involve combining USAP with diversity-based sampling strategies or extending the "maximizing expected metrics" framework to other complex measures like AUC or F-measure.

Conclusion

USAP represents a significant step forward in active learning for action recognition. By directly maximizing the metric that matters—Average Precision—it reduces the labeling burden while achieving superior model performance across diverse video benchmarks.

Summary updated

BibTeX
@inproceedings{wang2018uncertainty,
  author = {Hanmo Wang and Xiaojun Chang and Lei Shi and Yi Yang and Yi-Dong Shen},
  title = {{Uncertainty Sampling for Action Recognition via Maximizing Expected Average Precision}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/134},
}
Figure