Annotation-Efficient Untrimmed Video Action Recognition

Yixiong Zou, Shanghang Zhang, Guangyao Chen, Yonghong Tian, Kurt Keutzer, José M. F. Moura

Abstract

Deep learning has achieved great success in recognizing video actions, but the collection and annotation of training data remain laborious. This difficulty stems from two main factors: the massive volume of annotated data required and the time-consuming nature of temporally locating each action. While few-shot learning and untrimmed video recognition address these issues separately, few works handle both simultaneously. We introduce "Annotation-Efficient Video Recognition," a framework designed to reduce requirements for both sample quantity and temporal location. We categorize background segments into informative (IBG) and non-informative (NBG) and propose an open-set detection method, contrastive learning, and a self-weighting mechanism to distinguish them. Experiments on ActivityNet v1.2 and v1.3 demonstrate the effectiveness of our approach.

The Challenge: The Quantity and Granularity Gaps

Modern video recognition models are notoriously data-hungry. Research typically faces two bottlenecks:

  1. The Quantity Gap: Models often require thousands of examples to learn a new action. Few-shot learning attempts to mitigate this but usually assumes "trimmed" videos where only the action is shown.
  2. The Granularity Gap: Manually marking the start and end times (temporal annotation) in long, untrimmed videos is extremely labor-intensive.
The landscape of video recognition research, positioning the proposed 'Annotation-Efficient' approach relative to existing few-shot and untrimmed video methods.

A critical, often overlooked challenge is Overlapped Background. A background segment in one class (e.g., people running in a "Soccer" video) might be the primary action for another class ("Jogging"). If a model is forced to treat that background as "trash" during training, it loses the ability to recognize it as a valid action later.

Methodology: Learning from Weak Supervision

To address these gaps, we propose a three-pronged approach that learns from untrimmed videos using only video-level labels. This bypasses the need for manual frame-by-frame trimming.

The labor-intensive process of temporal annotation, where untrimmed videos are manually cut into trimmed segments for training.

Categorizing Background (BG)

Instead of treating all non-target segments as noise, we distinguish between:

  • Non-Informative BG (NBG): Static logos, intro/outro credits, or blank screens.
  • Informative BG (IBG): Segments containing humans, objects, or motions that aren't the target action but contain useful semantic information.

Open-Set Detection and Contrastive Learning

Since we lack timestamps, we use open-set detection logic to generate pseudo-labels. Segments with high classification uncertainty are labeled as NBG, while high-confidence segments are labeled as Foreground (FG). Those in between are treated as IBG. A contrastive loss then pulls NBG segments together in the feature space while pushing them away from the combined IBG+FG group, forcing the model to learn what makes a segment "informative."

Self-Weighting Mechanism

Standard models use a global "attention" network to decide which parts of a video matter. We replace this with a Self-Weighting mechanism that compares every segment in a video to that specific video's pseudo-labeled background. This handles the massive diversity of backgrounds much better than a one-size-fits-all global network.

Experimental Results

We evaluated our approach on ActivityNet v1.2 (100 classes) and v1.3 (200 classes). Crucially, we did not use temporal annotations for base training, relying solely on video-level labels.

  • Classification: On ActivityNet v1.3 (5-way 1-shot), our model achieved 73.39% accuracy, a significant jump over the standard MatchingNet baseline (~71%).
  • Detection: Despite being trained with Weak Supervision (no timestamps), the model achieved a [email protected] of 30.24 on ActivityNet v1.2. This is remarkably close to (and in 5-shot scenarios, better than) models that use Full Supervision (manually labeled timestamps).

Ablation studies confirmed that the contrastive learning module was the single biggest contributor to performance, proving that learning to distinguish "informative" content is key to few-shot success.

Conclusion and Future Work

By recognizing that background is not "trash" and leveraging uncertainty as a signal, our framework significantly reduces the annotation burden for video recognition. Future work will focus on reducing the requirement for trimmed videos in the "support set" (the few examples used to define a new class) and addressing potential noise in the initial pseudo-labeling phase.

Summary updated

Abstract

Deep learning has achieved great success in recognizing video actions, but the collection and annotation of training data are still quite laborious, which mainly lies in two aspects: (1) the amount of required annotated data is large; (2) temporally annotating the location of each action is time-consuming. Works such as few-shot learning or untrimmed video recognition have been proposed to handle either one aspect or the other. However, very few existing works can handle both issues simultaneously. In this paper, we target a new problem, Annotation-Efficient Video Recognition, to reduce the requirement of annotations for both large amount of samples and the action location. Such problem is challenging due to two aspects: (1) the untrimmed videos only have weak supervision; (2) video segments not relevant to current actions of interests (background, BG) could contain actions of interests (foreground, FG) in novel classes, which is a widely existing phenomenon but has rarely been studied in few-shot untrimmed video recognition. To achieve this goal, by analyzing the property of BG, we categorize BG into informative BG (IBG) and non-informative BG (NBG), and we propose (1) an open-set detection based method to find the NBG and FG, (2) a contrastive learning method to learn IBG and distinguish NBG in a self-supervised way, and (3) a self-weighting mechanism for the better distinguishing of IBG and FG. Extensive experiments on ActivityNet v1.2 and ActivityNet v1.3 verify the rationale and effectiveness of the proposed methods.
BibTeX
@inproceedings{zou2021annotation,
  author = {Yixiong Zou and Shanghang Zhang and Guangyao Chen and Yonghong Tian and Kurt Keutzer and José M. F. Moura},
  title = {{Annotation-Efficient Untrimmed Video Action Recognition}},
  booktitle = {ACM Multimedia},
  year = {2021},
  doi = {10.1145/3474085.3475197},
}
Figure