Abstract
Action recognition is computationally expensive. In this paper, we address the problem of frame selection to improve the accuracy of action recognition. In particular, we show that selecting good frames helps in action recognition performance even in the trimmed videos domain. Recent work has successfully leveraged frame selection for long, untrimmed videos, where much of the content is not relevant, and easy to discard. In this work, however, we focus on the more standard short, trimmed action recognition problem. We argue that good frame selection can not only reduce the computational cost of action recognition but also increase the accuracy by getting rid of frames that are hard to classify. In contrast to previous work, we propose a method that instead of selecting frames by considering one at a time, considers them jointly. This results in a more efficient selection, where good frames are more effectively distributed over the video, like snapshots that tell a story. We call the proposed frame selection SMART and we test it in combination with different backbone architectures and on multiple benchmarks (Kinetics, Something-something, UCF101). We show that the SMART frame selection consistently improves the accuracy compared to other frame selection strategies while reducing the computational cost by a factor of 4 to 10 times. Additionally, we show that when the primary goal is recognition performance, our selection strategy can improve over recent state-of-the-art models and frame selection strategies on various benchmarks (UCF101, HMDB51, FCVID, and ActivityNet).
Introduction
Action recognition—identifying what is happening in a video—is notoriously computationally expensive. Traditionally, researchers have tried to improve accuracy by building larger, deeper neural networks. However, videos are inherently redundant; many frames look nearly identical, and some are simply "noise" (blurred or occluded) that can actually confuse a classifier.
While previous work has successfully used frame selection for long, "untrimmed" videos, it remains a difficult challenge for "trimmed" videos. These are short, 3-to-10-second clips where the action is dense. In these cases, "good" frames are often very close together, making it hard to pick a representative sample that captures the full temporal "story" of the action.
The SMART Framework
The authors propose SMART (Sampling through Multi-frame Attention and Relations in Time), a lightweight selection network that acts as a gatekeeper for more expensive "backbone" models (like ResNet-152).
The system consists of three main components:
- Enriched Feature Representation: SMART uses MobileNet for visual features and GloVe embeddings for language features. It averages word embeddings of the top 10 ImageNet classes predicted for a frame to understand context (e.g., seeing "water" and "paddle" helps identify "kayaking").
- The Single-frame Selector: A simple MLP that predicts how "useful" an individual frame is based on how confident a high-end "oracle" model would be in classifying it.
- The Global Selector: This component considers pairs of frames and uses an Attention Module combined with an LSTM to understand temporal relationships. Instead of picking frames in isolation, it looks for a sequence of frames that "tell a story" together.
Key Insights
- The "Joint" Advantage: Most selectors pick frames one by one. SMART’s global selector considers how frames relate to one another. This ensures the selected frames are diverse and representative of the beginning, middle, and end of an action.
- Language as a Signal: Semantic information from GloVe embeddings provides a "prior" that helps the model focus on relevant objects within the frame.
- The "Sweet Spot" Phenomenon: Accuracy doesn't just plateau as frames are added—it can actually decrease. There is a "sweet spot" (often around 26 frames) where the model has enough information but hasn't yet been "polluted" by hard-to-classify or redundant frames.
Experimental Results
The authors evaluated SMART across a wide variety of benchmarks including Something-something-v2, Kinetics, UCF101, HMDB51, ActivityNet, and FCVID.
- Efficiency: SMART reduced computational costs (measured in GFLOPs) by 4x to 10x compared to processing all frames.
- Accuracy Boost: On the Something-something-v2 dataset, SMART achieved 60.8% accuracy using an Inception-v3 backbone, outperforming the "All frames" baseline of 58.8%.
- State-of-the-Art (SOTA): SMART helped achieve new SOTA results on UCF101 (98.6%) and HMDB51 (84.3%).
Conclusion
SMART demonstrates that in action recognition, "less is more." By intelligently selecting a subset of frames, we can improve both efficiency and accuracy. Future work may explore adaptive budgets for frame selection and even more lightweight feature extractors to further maximize efficiency gains.
Summary updated