TL;DR
This paper introduces Annotation-Efficient Video Recognition, a framework that trains action recognition models using significantly less data and effort. By distinguishing between "informative" background (objects/motions that might be actions in other contexts) and "non-informative" background (like logos or credits), the authors enable models to learn from untrimmed videos with only video-level labels, significantly outperforming previous few-shot learning methods.
What Problem Does This Paper Address?
Deep learning for video recognition is notoriously "data-hungry" and "label-heavy." There are two main bottlenecks:
- The Quantity Gap: Models usually need thousands of examples to learn a new action. "Few-shot learning" tries to fix this but usually assumes the videos are already "trimmed" (cut to show only the action).
- The Granularity Gap: Annotating exactly when an action starts and ends in a long, untrimmed video is incredibly time-consuming. "Weakly-supervised" methods try to learn from video-level labels (e.g., "this video contains swimming") without timestamps, but they struggle when the background is cluttered.
The authors identify a specific "hidden" challenge: Overlapped Background. A background segment in a "Soccer" video (like people running) might actually be the foreground action for a "Jogging" class. If a model is forced to treat that background as "trash" during training, it loses the ability to recognize it as a valid action later.
Methods
The authors propose a three-pronged approach to handle untrimmed videos with only weak supervision:
- Categorizing Background (BG): Instead of treating all non-action segments as "noise," they split them into:
- Non-Informative BG (NBG): Static logos, intro/outro credits, or blank screens.
- Informative BG (IBG): Segments containing humans, objects, or motions that aren't the target action but contain useful semantic information.
- Open-Set Detection for Pseudo-Labeling: Since they don't have timestamps, the model uses "open-set detection" logic. Segments that the classifier is very uncertain about (where the probability distribution is nearly flat) are pseudo-labeled as NBG. Segments with high confidence are labeled FG (Foreground), and those in the middle are IBG.
- Contrastive Learning: To help the model capture the "essence" of an action, they use a contrastive loss. It pulls different NBG segments together in the feature space while pushing NBG away from the combined (IBG + FG) group. This forces the model to learn what makes a segment "informative."
- Self-Weighting Mechanism: Standard models use a global "attention" network to decide which parts of a video matter. This paper replaces that with a Self-Weighting mechanism: it compares every segment in a video to that specific video's pseudo-labeled background. This handles the massive diversity of backgrounds much better than a one-size-fits-all global network.
Datasets
The researchers evaluated their approach on two major benchmarks:
- ActivityNet v1.2: 100 action classes. They used 80 for "base" training and 20 for "novel" few-shot testing.
- ActivityNet v1.3: 200 action classes (160 base / 40 novel). Unlike many previous works, they do not use the temporal (start/end) annotations for the base training sets, relying only on video-level class labels.
Key Insights and Ideas
- Background is not "Trash": The most important "aha moment" is that forcing the model to ignore informative background harms its ability to generalize. By allowing the model to "see" the IBG without labeling it as the target action, it retains a richer vocabulary of motions and objects.
- Uncertainty is a Signal: Using classification uncertainty (via open-set detection) is a clever way to automatically find "boring" parts of a video (NBG) without a human having to mark them.
- Local Context over Global Rules: The self-weighting mechanism acknowledges that "background" is relative. What is background in one video might be very different from background in another; comparing a segment to its own video's context is more robust than using a global background model.
Key Results
The proposed method consistently outperformed state-of-the-art (SOTA) baselines in both action classification and temporal action detection:
- Classification: On ActivityNet v1.3 (5-way 1-shot), the model achieved 73.39% accuracy, a significant jump over the standard MatchingNet baseline (~71%).
- Detection: Even though the model was trained with Weak Supervision (no timestamps), it achieved a [email protected] of 30.24 on ActivityNet v1.2. This is remarkably close to (and in 5-shot scenarios, better than) models like F-PAD-ctrl that use Full Supervision (manually labeled timestamps).
- Ablation: The contrastive learning module was the single biggest contributor to performance, proving that learning to distinguish "informative" content is key to few-shot success.
Limitations and Future Work
- Support Set Trimming: While the model handles untrimmed videos for training and testing, the "support set" (the 1 or 5 examples used to define a new class) still requires trimmed videos. Reducing this requirement further is a logical next step.
- Pseudo-label Noise: The method relies on the model's own initial guesses to label NBG/IBG. If the initial model is very poor, these pseudo-labels could be noisy and lead to "the blind leading the blind" during training.
- Computational Overhead: Contrastive learning on video segments can be memory-intensive, which might limit the length of videos the model can process at once.
Summary updated