Look More but Care Less in Video Recognition

Yitian Zhang, Yue Bai, Huan Wang, Yi Xu, Yun Fu

Abstract

Video recognition models often face a trade-off between temporal coverage and computational efficiency. Sparse sampling can miss critical information, while dense processing is prohibitively expensive. We introduce the Ample and Focal Network (AFNet), a novel framework that processes all video frames through a lightweight "Ample" branch for global context while dynamically selecting salient frames for high-resolution processing in a "Focal" branch. This dual-pathway approach allows the model to "look" at more frames while "caring" less about redundant ones, achieving state-of-the-art efficiency on major benchmarks like Mini-Kinetics and Something-Something.

Introduction

Processing video with deep convolutional neural networks (CNNs) is computationally demanding because costs scale linearly with the number of frames. To maintain efficiency, standard practices involve sampling a small subset of frames (e.g., 8 or 12), which often leads to information loss. Previous dynamic sampling methods that discard frames entirely suffer from non-differentiable decision-making and permanent loss of context. AFNet addresses these challenges by ensuring no frame is completely ignored, instead varying the intensity of processing based on saliency.

Methodology

The core innovation of AFNet is the AF Module, which replaces standard convolutional blocks. It consists of three primary components:

  1. Ample Branch: A low-cost pathway that processes all input frames at reduced resolution and channel depth. It provides a continuous global context of the video.
  2. Focal Branch: A high-capacity pathway that operates on full-resolution features but only for a subset of frames.
  3. Navigation Module: Acting as the system's "brain," this module uses features from the Ample branch to generate a temporal mask. This mask determines which frames the Focal branch should process at each layer. The Gumbel-Softmax trick is employed to make this discrete selection differentiable.
Detailed architecture of the AF Module, illustrating the dual-branch structure, navigation module, and dynamic fusion mechanism.

Finally, Dynamic Fusion adaptively merges features from both branches, ensuring that even "skipped" frames contribute their basic context to the final representation.

Experimental Results

AFNet was evaluated on several diverse datasets, including Mini-Kinetics, ActivityNet-v1.3, Jester, and Something-Something (V1 & V2).

Efficiency and Accuracy

AFNet consistently demonstrates a superior accuracy-efficiency profile compared to existing state-of-the-art methods. On Mini-Kinetics, AFNet achieved 73.5% Top-1 accuracy using only 22.0 GFLOPs, outperforming competitors like AdaFocus and VideoIQ.

Comparison of accuracy vs. computational cost (GFLOPs) on the Mini-Kinetics dataset. AFNet shows a significant improvement in the efficiency-accuracy trade-off.

Key Insights

  • Noise Reduction: Selecting fewer frames in the Focal branch often improves accuracy, suggesting that dynamic selection acts as a filter for temporal noise.
  • Implicit Temporal Modeling: By selecting different frames across different layers, AFNet effectively implements a form of frame-wise attention, capturing temporal relationships without the need for expensive 3D convolutions.
  • Preservation: The Ample branch prevents the "all-or-nothing" risk of traditional frame-skipping, maintaining a baseline level of information for every frame.

Conclusion

AFNet provides a robust solution to temporal redundancy in video recognition. By combining a cheap global glimpse with expensive local focus, it achieves high performance with reduced computational overhead. While current implementations require specialized backbone training, the "Ample and Focal" philosophy holds promise for broader applications in multi-modal learning and high-resolution image analysis.

Summary updated

BibTeX
@inproceedings{zhang2022look,
  author = {Yitian Zhang and Yue Bai and Huan Wang and Yi Xu and Yun Fu},
  title = {{Look More but Care Less in Video Recognition}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=MDawWwJT2E},
}
Figure