Abstract
Long-tailed distribution in video datasets presents a significant challenge for recognition models, as frequent "head" classes often dominate training while rare "tail" classes suffer from poor representation. This problem is further complicated in multi-label video contexts where common and rare actions co-occur within the same sequence. We introduce MEID (Mixture-of-Experts with Internal Distillation), a framework designed to address these challenges through a dual-expert architecture and a novel feature distillation mechanism. MEID utilizes Complementary Frame Selection to focus on "hard" frames and transfers feature richness from head to tail classes via variance-based distillation. Our approach achieves state-of-the-art results on VideoLT and two new densely labeled benchmarks, CharadesLT and CharadesEgoLT.
Introduction
In real-world computer vision applications, action frequencies typically follow a long-tailed distribution. While a model might encounter thousands of instances of common actions like "walking," it may only see a handful of rare actions. This imbalance leads to models that are biased toward "head" classes and perform poorly on the "tail."
Video recognition introduces unique complexities to this problem. First, a single untrimmed video often contains multiple labels across the frequency spectrum (multi-label complexity). Second, the specific frames representing a rare action might be buried within a video dominated by common actions (frame-level imbalance). MEID addresses these issues by specializing model components at the frame level rather than the video level.
Methodology
The MEID framework consists of two primary innovations: a dual-expert strategy and an internal distillation process.
Dual-Expert Strategy with Complementary Frame Selection (CFS)
Instead of a single monolithic model, MEID employs a Mixture-of-Experts (MoE) approach:
- Expert 1: Trained on the original, imbalanced distribution. It uses an attention mechanism to identify the most "important" frames for classification.
- Expert 2: Trained on a re-balanced distribution. It utilizes Complementary Frame Selection (CFS) by inheriting the attention weights from Expert 1 and taking their reciprocal. This forces Expert 2 to ignore the "easy" frames already solved by Expert 1 and focus exclusively on the "hard" frames that contain rare or subtle actions.
- Motion Enhancement: Expert 2 also includes a motion extraction module to capture temporal nuances, which are often critical for distinguishing rare actions.
Internal Distillation
To improve the feature representations of rare classes, MEID introduces a Head-to-Tail Distillation Loss. While head classes have "rich" feature representations due to abundant data, tail classes often suffer from "collapsed" representations. MEID encourages tail-class features to match the statistical variance of head-class features. This transfers knowledge about feature complexity without forcing semantically different actions to look alike.
Evaluation and Results
The authors evaluated MEID on three benchmarks: VideoLT, and two new densely labeled datasets introduced in this work, CharadesLT and CharadesEgoLT. These new benchmarks are designed to reflect real-world multi-label challenges, featuring higher label density (~7-9 labels per video) compared to VideoLT.
MEID demonstrated significant performance gains:
- VideoLT: Reached 61.9% mAP using a ResNet-50 backbone, outperforming the previous state-of-the-art (MEDC) by 5.2%.
- Tail Class Performance: On VideoLT with a ResNet-101 backbone, performance on tail classes improved from 0.419 mAP to 0.652 mAP, a substantial relative gain.
- New Benchmarks: MEID established strong baselines on CharadesLT and CharadesEgoLT, consistently outperforming standard long-tail methods like LDAM and Mixup.
Conclusion
MEID demonstrates that frame-level assignment and feature variance distillation are key to tackling the long-tail problem in video recognition. By ensuring that experts focus on complementary information and that rare classes inherit the feature richness of common ones, the framework significantly improves recognition accuracy for under-represented actions. Future research may focus on end-to-end training and extending this approach to temporal action localization.
Summary updated