Abstract
Few-shot action recognition (FSAR) aims to classify novel video actions with minimal labeled examples. Current approaches often over-fit to task-specific matching, limiting their generalization to unseen categories. We present Task-Agnostic Self-Distillation (TASD), a framework that leverages a momentum-based teacher network to learn universal matching rules. By distilling multi-dimensional features and global pixel relations from a stable teacher ensemble, the student network develops robust, task-agnostic representations. TASD achieves state-of-the-art performance on Kinetics-CMN, UCF101, and HMDB51 benchmarks, particularly in challenging 1-shot scenarios.
Introduction
Few-shot action recognition (FSAR) is a critical challenge in computer vision, requiring models to identify new action classes from just one or a few examples. Traditional methods typically focus on task-specific matching, optimizing the comparison between query and support videos within a single, isolated episode. However, this narrow focus often prevents models from learning general features that define action similarity across diverse categories.
The TASD framework addresses this by shifting the objective from "how to match these specific videos" to "how to match videos in general." By encouraging the model to learn universal metric features, it enhances generalization to completely novel action types.
Methodology
The TASD framework introduces a teacher-student architecture designed for robust feature distillation.
Momentum-Based Self-Distillation
The system utilizes a Student network that learns from the current task and a Teacher network whose parameters are an Exponentially Moving Average (EMA) of the Student’s historical states. This EMA Teacher acts as a stable memory or ensemble of the training process, providing a global, task-agnostic perspective to guide the Student.
Multi-Dimension Distillation (MDD)
To capture the spatio-temporal complexity of video data, the MDD module extracts and aligns features across three dimensions: Temporal, Spatial, and Channel. Attention maps are used to ensure the Student mimics the Teacher’s focus on relevant video segments.
Global Pixel Relation Distillation (PR)
Using Graph Convolutional Networks (GloRe), the PR module models the relationship between foreground actions and background context. Distilling these relational features from the Teacher to the Student helps the model distinguish primary actions from environmental noise.
Results and Discussion
TASD was evaluated on Kinetics-CMN, UCF101, and HMDB51 using standard 5-way 1-shot and 5-shot protocols.
Performance Comparison
The method achieved state-of-the-art (SOTA) results in 1-shot scenarios:
- Kinetics-CMN: 76.5% accuracy (a 2.5% improvement over the previous best, MoLo).
- UCF101: 88.2% accuracy (a 2.2% improvement over previous SOTA).
- HMDB51: 61.1% accuracy.
In 5-shot settings, TASD remained highly competitive, matching or exceeding specialized methods like SloshNet and TRX.
Evolution of Attention
Visualization of attention maps reveals that as training progresses, the Teacher network becomes increasingly concentrated on universal, task-agnostic features, which then guides the Student to ignore task-specific noise.
A key finding was that Cross-Set Distillation (distilling knowledge between different videos of the same class) outperformed Self-Set Distillation. This suggests that the model successfully learns category-level semantics rather than instance-specific visual patterns.
Conclusion
Task-Agnostic Self-Distillation (TASD) effectively bridges the gap between task-specific matching and general action recognition. While it excels in 1-shot scenarios, future work may explore dynamic balancing of task-agnostic and task-specific information for many-shot settings. Additionally, optimizing the computational overhead of the teacher-student framework remains an area for further research.
Summary updated