SMART Frame Selection for Action Recognition

Shreyank N. Gowda, Marcus Rohrbach, Laura Sevilla-Lara

Abstract

SMART (Multi-frame Attention and Relations in Time) is a novel frame selection strategy that improves action recognition by considering frames jointly rather than in isolation. By picking a diverse set of "story-telling" frames, the method achieves higher accuracy than using the full video while reducing computational costs by 4 to 10 times.

Introduction

Action recognition in video is computationally expensive because processing every frame with a deep neural network requires massive resources. While researchers have developed "frame selection" methods to skip redundant data, most focus on untrimmed videos—long clips where most of the content is irrelevant background noise.

Selecting frames in short, trimmed videos (usually just a few seconds long) is much harder. In these clips, "good" (discriminative) frames are often clustered closely together. Existing methods typically evaluate frames one-by-one, which often leads to selecting redundant frames that don't capture the full progression of an action. The authors argue that a good selection should act like a "snapshot that tells a story," requiring a global understanding of the video's temporal structure.

Methodology

The authors propose the SMART selection network, which uses a two-stream approach to score frame importance:

  1. Single-frame Selector: A lightweight Multi-Layer Perceptron (MLP) that predicts how useful a single frame is for classification. It is trained using an "oracle"—the confidence scores of a heavy, state-of-the-art model—as ground truth.
  2. Global Selector: This is the core innovation. It uses an attention mechanism and a Relational Network (incorporating an LSTM) to examine frames in relation to one another. By looking at pairs of frames, the model learns to identify temporal changes and ensure the selected frames provide a diverse representation of the action.
The SMART architecture overview, showing the dual-stream selection process.

To keep the selector efficient, the system uses MobileNet for visual features and incorporates language features. By taking the top-10 ImageNet class predictions for a frame and embedding them using GloVe, the model gains semantic context (e.g., identifying a "paddle" or "water") to better judge a frame's relevance to specific actions.

Key Insights

  • Joint Selection vs. Individual Selection: SMART treats frame selection as a set-discovery problem. By ensuring chosen frames complement each other, it avoids the redundancy common in individual frame scoring.
  • The "Sweet Spot" of Information: Accuracy can actually drop if too many frames are used, as "bad" frames (motion blur, occlusions) pollute the prediction. SMART identifies the optimal subset that maximizes signal-to-noise.
  • Oracle Distillation: Training a small network to predict the confidence of a much larger network effectively bridges the gap between efficiency and accuracy.

Key Results

SMART was evaluated on benchmarks including Something-something-v2, Kinetics, UCF101, and HMDB51. It consistently outperformed standard uniform and random sampling strategies.

Top-1 Accuracy vs. Number of Frames, demonstrating SMART's superiority and the 'sweet spot' effect.
  • Efficiency: SMART reduces computational cost (GFLOPs) by 4x to 10x compared to processing full videos.
  • Accuracy Boost: On UCF101, SMART achieved 75.3% accuracy using only 26 frames, outperforming the "All Frames" baseline (74.6%).
  • State-of-the-Art: When paired with high-end models like I3D or TSN, SMART helped achieve state-of-the-art results on HMDB51 (84.3%) and UCF101 (98.6%).

Conclusion

SMART demonstrates that a global, relational approach to frame selection is highly effective for action recognition. While the current pipeline requires a full video scan for initial feature extraction, future work aims to integrate the selector and backbone classifier end-to-end to further optimize performance and efficiency.

Summary updated

Abstract

Video classification is computationally expensive. In this paper, we address theproblem of frame selection to reduce the computational cost of video classification.Recent work has successfully leveraged frame selection for long, untrimmed videos,where much of the content is not relevant, and easy to discard. In this work, however,we focus on the more standard short, trimmed video classification problem. Weargue that good frame selection can not only reduce the computational cost of videoclassification but also increase the accuracy by getting rid of frames that are hard toclassify. In contrast to previous work, we propose a method that instead of selectingframes by considering one at a time, considers them jointly. This results in a moreefficient selection, where “good" frames are more effectively distributed over thevideo, like snapshots that tell a story. We call the proposed frame selection SMARTand we test it in combination with different backbone architectures and on multiplebenchmarks (Kinetics [5], Something-something [14], UCF101 [31]). We showthat the SMART frame selection consistently improves the accuracy compared toother frame selection strategies while reducing the computational cost by a factorof 4 to 10 times. Additionally, we show that when the primary goal is recognitionperformance, our selection strategy can improve over recent state-of-the-art modelsand frame selection strategies on various benchmarks (UCF101, HMDB51 [21],FCVID [17], and ActivityNet [4]).
BibTeX
@inproceedings{gowda2021smart,
  author = {Shreyank N. Gowda and Marcus Rohrbach and Laura Sevilla-Lara},
  title = {{SMART Frame Selection for Action Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i2.16235},
}

Loading related content...

Figure