Abstract
Deep learning has achieved great success in recognizing video actions, but the collection and annotation of training data remain laborious. This difficulty stems from two main factors: the massive volume of annotated data required and the time-consuming nature of temporally locating each action. While few-shot learning and untrimmed video recognition address these issues separately, few works handle both simultaneously. We introduce "Annotation-Efficient Video Recognition," a framework designed to reduce requirements for both sample quantity and temporal location. We categorize background segments into informative (IBG) and non-informative (NBG) and propose an open-set detection method, contrastive learning, and a self-weighting mechanism to distinguish them. Experiments on ActivityNet v1.2 and v1.3 demonstrate the effectiveness of our approach.
The Challenge: The Quantity and Granularity Gaps
Modern video recognition models are notoriously data-hungry. Research typically faces two bottlenecks:
- The Quantity Gap: Models often require thousands of examples to learn a new action. Few-shot learning attempts to mitigate this but usually assumes "trimmed" videos where only the action is shown.
- The Granularity Gap: Manually marking the start and end times (temporal annotation) in long, untrimmed videos is extremely labor-intensive.
A critical, often overlooked challenge is Overlapped Background. A background segment in one class (e.g., people running in a "Soccer" video) might be the primary action for another class ("Jogging"). If a model is forced to treat that background as "trash" during training, it loses the ability to recognize it as a valid action later.
Methodology: Learning from Weak Supervision
To address these gaps, we propose a three-pronged approach that learns from untrimmed videos using only video-level labels. This bypasses the need for manual frame-by-frame trimming.
Categorizing Background (BG)
Instead of treating all non-target segments as noise, we distinguish between:
- Non-Informative BG (NBG): Static logos, intro/outro credits, or blank screens.
- Informative BG (IBG): Segments containing humans, objects, or motions that aren't the target action but contain useful semantic information.
Open-Set Detection and Contrastive Learning
Since we lack timestamps, we use open-set detection logic to generate pseudo-labels. Segments with high classification uncertainty are labeled as NBG, while high-confidence segments are labeled as Foreground (FG). Those in between are treated as IBG. A contrastive loss then pulls NBG segments together in the feature space while pushing them away from the combined IBG+FG group, forcing the model to learn what makes a segment "informative."
Self-Weighting Mechanism
Standard models use a global "attention" network to decide which parts of a video matter. We replace this with a Self-Weighting mechanism that compares every segment in a video to that specific video's pseudo-labeled background. This handles the massive diversity of backgrounds much better than a one-size-fits-all global network.
Experimental Results
We evaluated our approach on ActivityNet v1.2 (100 classes) and v1.3 (200 classes). Crucially, we did not use temporal annotations for base training, relying solely on video-level labels.
- Classification: On ActivityNet v1.3 (5-way 1-shot), our model achieved 73.39% accuracy, a significant jump over the standard MatchingNet baseline (~71%).
- Detection: Despite being trained with Weak Supervision (no timestamps), the model achieved a [email protected] of 30.24 on ActivityNet v1.2. This is remarkably close to (and in 5-shot scenarios, better than) models that use Full Supervision (manually labeled timestamps).
Ablation studies confirmed that the contrastive learning module was the single biggest contributor to performance, proving that learning to distinguish "informative" content is key to few-shot success.
Conclusion and Future Work
By recognizing that background is not "trash" and leveraging uncertainty as a signal, our framework significantly reduces the annotation burden for video recognition. Future work will focus on reducing the requirement for trimmed videos in the "support set" (the few examples used to define a new class) and addressing potential noise in the initial pseudo-labeling phase.
Summary updated