Search-Map-Search: A Frame Selection Paradigm for Action Recognition

Mingjun Zhao, Yakun Yu, Xiaoli Wang, Lei Yang, Di Niu

Introduction

Action recognition in video is a computationally intensive task. While a video may span thousands of frames, identifying the core action—such as "diving" or "making a sandwich"—typically requires only a small subset of informative frames. Efficiently selecting these frames is critical for reducing the overhead of processing redundant or irrelevant temporal data.

Existing approaches generally follow two strategies:

  1. Per-frame importance: Models score frames individually and select the top $n$. However, this often leads to redundancy, as the model fails to account for temporal interactions (e.g., selecting multiple nearly identical frames of a single motion).
  2. Reinforcement Learning (RL): Agents are trained to "hop" through videos to find key frames. RL is notoriously difficult to train and operates as a sequential Markov Decision Process. Once an agent moves past a frame, it cannot revisit or adjust previous selections based on future context.

To address these limitations, the authors introduce Search-Map-Search (SMS), a framework that treats frame selection as a global optimization problem rather than a sequential decision process.

The Search-Map-Search Framework

The SMS paradigm consists of three distinct stages designed to find, learn, and then infer the optimal combination of frames for a given video.

The three-stage Search-Map-Search (SMS) framework: Stage 1 searches for optimal combinations, Stage 2 learns a mapping function, and Stage 3 performs fast inference.
  1. Stage 1: Search (Training Only): For every training video, a Hierarchical Guided Local Search identifies the best combination of $n$ frames that minimizes the loss of a pre-trained action recognition model. To manage the search space, the process begins at a coarse "clip" level before refining at the "frame" level, creating a "gold standard" target for supervised learning.
  2. Stage 2: Map (Learning): A Feature Mapping Function, implemented with Transformer layers, is trained to project the features of the entire input video into a target feature vector. This vector represents the "optimal" frame combination identified in Stage 1. The model effectively learns the "ideal" summary of a video in feature space.
  3. Stage 3: Search (Inference): For unseen videos, the mapping function predicts the ideal feature vector. A final search identifies the actual frames whose combined features most closely match this predicted ideal using cosine distance. This search is performed in a low-dimensional space, ensuring high speed.

Experimental Setup

The SMS framework was evaluated on three major action recognition benchmarks:

  • ActivityNet V1.3: Large-scale untrimmed videos (avg. 117s).
  • FCVID: Large-scale untrimmed dataset (avg. 167s).
  • UCF101: Short, trimmed videos (avg. 7.2s).

Key Insights

  • Global Optimization: By optimizing for a combination of frames, SMS naturally accounts for diversity and inter-frame interactions, avoiding the redundancy inherent in per-frame methods.
  • Representation Learning: Predicting the representation of the best frames is a more stable learning task than predicting discrete frame indices, which is a complex combinatorial problem.
  • Hierarchical Efficiency: The coarse-to-fine search strategy makes it feasible to find ground-truth selections even in long videos with massive search spaces.
  • Model Agnostic: SMS serves as a "plug-and-play" front-end, compatible with both 2D CNNs (e.g., ResNet) and 3D spatio-temporal transformers (e.g., TimeSFormer).

Results and Discussion

SMS consistently outperforms state-of-the-art frame selection methods, including SCSampler, AdaFrame, and SMART, across various metrics.

Comparison of SMS against state-of-the-art methods on the accuracy-computation Pareto front. SMS (cyan) achieves higher precision with lower computational cost (GFLOPs).
  • Accuracy Gains: On ActivityNet, using only 8 frames, SMS achieved 83.72% mAP, representing a +6.38% improvement over uniform sampling.
  • Efficiency: SMS provides a superior Pareto front for the accuracy-vs-computation tradeoff. As shown in the results, SMS with 8 frames often outperforms competing methods that utilize 16 or more frames.
  • Generalizability: When integrated with the TimeSFormer model, SMS provided a +1.86% mAP boost over standard sampling on ActivityNet, reaching 91.97% mAP.
  • Data Efficiency: The framework remains effective even with limited data; SMS achieves competitive results when trained on only 10% of the available labels.

Conclusion

While SMS demonstrates significant gains in accuracy and efficiency, it does introduce some training overhead due to the search process in Stage 1. Additionally, the current implementation assumes a fixed frame count, and performance is tied to the quality of the pre-trained feature extractor. Future work may explore adaptive frame counts based on video complexity to further optimize the selection process.

Summary updated

Abstract

Despite the success of deep learning in video understanding tasks, processing every frame in a video is computationally expensive and often unnecessary in real-time applications. Frame selection aims to extract the most informative and representative frames to help a model better understand video content. Existing frame selection methods either individually sample frames based on per-frame importance prediction, without considering interaction among frames, or adopt reinforcement learning agents to find representative frames in succession, which are costly to train and may lead to potential stability issues. To overcome the limitations of existing methods, we propose a Search-Map-Search learning paradigm which combines the advantages of heuristic search and supervised learning to select the best combination of frames from a video as one entity. By combining search with learning, the proposed method can better capture frame interactions while incurring a low inference overhead. Specifically, we first propose a hierarchical search method conducted on each training video to search for the optimal combination of frames with the lowest error on the downstream task. A feature mapping function is then learned to map the frames of a video to the representation of its target optimal frame combination. During inference, another search is performed on an unseen video to select a combination of frames whose feature representation is close to the projected feature representation. Extensive experiments based on several action recognition benchmarks demonstrate that our frame selection method effectively improves performance of action recognition models, and significantly outperforms a number of competitive baselines.
BibTeX
@inproceedings{zhao2023search,
  author = {Mingjun Zhao and Yakun Yu and Xiaoli Wang and Lei Yang and Di Niu},
  title = {{Search-Map-Search: A Frame Selection Paradigm for Action Recognition}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure