Annotation-Efficient Untrimmed Video Action Recognition

Yixiong Zou, Shanghang Zhang, Guangyao Chen, Yonghong Tian, Kurt Keutzer, José M. F. Moura

TL;DR

This paper introduces Annotation-Efficient Video Recognition, a framework that trains action recognition models using significantly less data and effort. By distinguishing between "informative" background (objects/motions that might be actions in other contexts) and "non-informative" background (like logos or credits), the authors enable models to learn from untrimmed videos with only video-level labels, significantly outperforming previous few-shot learning methods.

What Problem Does This Paper Address?

Deep learning for video recognition is notoriously "data-hungry" and "label-heavy." There are two main bottlenecks:

  1. The Quantity Gap: Models usually need thousands of examples to learn a new action. "Few-shot learning" tries to fix this but usually assumes the videos are already "trimmed" (cut to show only the action).
  2. The Granularity Gap: Annotating exactly when an action starts and ends in a long, untrimmed video is incredibly time-consuming. "Weakly-supervised" methods try to learn from video-level labels (e.g., "this video contains swimming") without timestamps, but they struggle when the background is cluttered.

The authors identify a specific "hidden" challenge: Overlapped Background. A background segment in a "Soccer" video (like people running) might actually be the foreground action for a "Jogging" class. If a model is forced to treat that background as "trash" during training, it loses the ability to recognize it as a valid action later.

Methods

The authors propose a three-pronged approach to handle untrimmed videos with only weak supervision:

  • Categorizing Background (BG): Instead of treating all non-action segments as "noise," they split them into:
  • Non-Informative BG (NBG): Static logos, intro/outro credits, or blank screens.
  • Informative BG (IBG): Segments containing humans, objects, or motions that aren't the target action but contain useful semantic information.
  • Open-Set Detection for Pseudo-Labeling: Since they don't have timestamps, the model uses "open-set detection" logic. Segments that the classifier is very uncertain about (where the probability distribution is nearly flat) are pseudo-labeled as NBG. Segments with high confidence are labeled FG (Foreground), and those in the middle are IBG.
  • Contrastive Learning: To help the model capture the "essence" of an action, they use a contrastive loss. It pulls different NBG segments together in the feature space while pushing NBG away from the combined (IBG + FG) group. This forces the model to learn what makes a segment "informative."
  • Self-Weighting Mechanism: Standard models use a global "attention" network to decide which parts of a video matter. This paper replaces that with a Self-Weighting mechanism: it compares every segment in a video to that specific video's pseudo-labeled background. This handles the massive diversity of backgrounds much better than a one-size-fits-all global network.

Datasets

The researchers evaluated their approach on two major benchmarks:

  • ActivityNet v1.2: 100 action classes. They used 80 for "base" training and 20 for "novel" few-shot testing.
  • ActivityNet v1.3: 200 action classes (160 base / 40 novel). Unlike many previous works, they do not use the temporal (start/end) annotations for the base training sets, relying only on video-level class labels.

Key Insights and Ideas

  • Background is not "Trash": The most important "aha moment" is that forcing the model to ignore informative background harms its ability to generalize. By allowing the model to "see" the IBG without labeling it as the target action, it retains a richer vocabulary of motions and objects.
  • Uncertainty is a Signal: Using classification uncertainty (via open-set detection) is a clever way to automatically find "boring" parts of a video (NBG) without a human having to mark them.
  • Local Context over Global Rules: The self-weighting mechanism acknowledges that "background" is relative. What is background in one video might be very different from background in another; comparing a segment to its own video's context is more robust than using a global background model.

Key Results

The proposed method consistently outperformed state-of-the-art (SOTA) baselines in both action classification and temporal action detection:

  • Classification: On ActivityNet v1.3 (5-way 1-shot), the model achieved 73.39% accuracy, a significant jump over the standard MatchingNet baseline (~71%).
  • Detection: Even though the model was trained with Weak Supervision (no timestamps), it achieved a [email protected] of 30.24 on ActivityNet v1.2. This is remarkably close to (and in 5-shot scenarios, better than) models like F-PAD-ctrl that use Full Supervision (manually labeled timestamps).
  • Ablation: The contrastive learning module was the single biggest contributor to performance, proving that learning to distinguish "informative" content is key to few-shot success.

Limitations and Future Work

  • Support Set Trimming: While the model handles untrimmed videos for training and testing, the "support set" (the 1 or 5 examples used to define a new class) still requires trimmed videos. Reducing this requirement further is a logical next step.
  • Pseudo-label Noise: The method relies on the model's own initial guesses to label NBG/IBG. If the initial model is very poor, these pseudo-labels could be noisy and lead to "the blind leading the blind" during training.
  • Computational Overhead: Contrastive learning on video segments can be memory-intensive, which might limit the length of videos the model can process at once.

Summary updated

Abstract

Deep learning has achieved great success in recognizing video actions, but the collection and annotation of training data are still quite laborious, which mainly lies in two aspects: (1) the amount of required annotated data is large; (2) temporally annotating the location of each action is time-consuming. Works such as few-shot learning or untrimmed video recognition have been proposed to handle either one aspect or the other. However, very few existing works can handle both issues simultaneously. In this paper, we target a new problem, Annotation-Efficient Video Recognition, to reduce the requirement of annotations for both large amount of samples and the action location. Such problem is challenging due to two aspects: (1) the untrimmed videos only have weak supervision; (2) video segments not relevant to current actions of interests (background, BG) could contain actions of interests (foreground, FG) in novel classes, which is a widely existing phenomenon but has rarely been studied in few-shot untrimmed video recognition. To achieve this goal, by analyzing the property of BG, we categorize BG into informative BG (IBG) and non-informative BG (NBG), and we propose (1) an open-set detection based method to find the NBG and FG, (2) a contrastive learning method to learn IBG and distinguish NBG in a self-supervised way, and (3) a self-weighting mechanism for the better distinguishing of IBG and FG. Extensive experiments on ActivityNet v1.2 and ActivityNet v1.3 verify the rationale and effectiveness of the proposed methods.
BibTeX
@inproceedings{zou2021annotation,
  author = {Yixiong Zou and Shanghang Zhang and Guangyao Chen and Yonghong Tian and Kurt Keutzer and José M. F. Moura},
  title = {{Annotation-Efficient Untrimmed Video Action Recognition}},
  booktitle = {ACM Multimedia},
  year = {2021},
  doi = {10.1145/3474085.3475197},
}

Loading related content...

Figure