Temporal Sequence Distillation: Towards Few-Frame Action Recognition in Videos

Zhaoyang Zhang, Zhanghui Kuang, Ping Luo, Litong Feng, Wei Zhang

Abstract

Video Analytics Software as a Service (VA SaaS) is rapidly expanding, typically involving lightweight clients transmitting data to cloud-based servers. However, limited and expensive transmission bandwidth creates a significant bottleneck. This paper introduces Temporal Sequence Distillation (TSD), a novel approach for few-frame action recognition. TSD distills long video sequences into very short, representative ones for transmission. By training end-to-end with 3D CNNs, TSD learns compact temporal and spatial representations. On the Kinetics dataset, the TSD+I3D framework achieves comparable accuracy to state-of-the-art models while requiring only 50% of the frames, significantly reducing both bandwidth and cloud computation costs.

The Challenge: Bandwidth in Video Analytics SaaS

The growth of VA SaaS faces a major hurdle: the "client-cloud" gap. Lightweight clients, such as mobile devices or smart cameras, must send video data to powerful cloud servers for complex tasks like action recognition. While 3D Convolutional Neural Networks (3D CNNs) are highly effective, they typically require dense frame stacks (often 64 or more) to capture sufficient temporal context. Transmitting such high-resolution, high-frame-rate video is bandwidth-intensive and costly.

The system architecture of Video Analytics SaaS, highlighting the transmission bottleneck between client and cloud.

To bridge this gap, we propose few-frame action recognition, aiming to maintain high recognition accuracy while accessing only a minimal number of frames during both training and testing.

Temporal Sequence Distillation (TSD)

TSD is a lightweight "plug-and-play" module designed to be deployed on the client side. Instead of simple random or uniform sampling, TSD learns to combine a long sequence into a very short, "distilled" one.

The architecture consists of a coarse feature extractor (MobileNet) and a TSD block that calculates a transformation matrix. The resulting distilled frames are linear combinations of the input frames, effectively acting as a "weighted average" optimized for recognition performance. This process often results in a "ghosting" or motion-blur effect in the distilled frames, which serves to encode motion information directly into the static representation.

Key Results and Performance

Extensive experiments on the Kinetics and UCF101 datasets demonstrate that TSD significantly improves efficiency without sacrificing accuracy. TSD identifies and merges the most informative parts of a video, eliminating the redundancy inherent in high-frame-rate clips.

Accuracy vs. Frame number on the Kinetics dataset, comparing standard I3D with the I3D+TSD module.

Key findings include:

  • Efficiency Gains: On Kinetics, I3D+TSD achieved 70.7% accuracy using only 60 frames, whereas the baseline I3D required 250 frames to reach a similar 71.1%.
  • Low-Frame Superiority: When restricted to a very small budget (e.g., 5 frames), TSD improved I3D's accuracy on Kinetics by 29.2% compared to standard uniform sampling.
  • Cloud Speedup: By reducing the frame count, TSD cut cloud-side computation (FLOPs) by approximately 50% and increased processing speeds by 45%–48%.

Conclusion

Temporal Sequence Distillation offers a practical solution for the bandwidth constraints of modern VA SaaS. By learning to distill motion and spatial features into a compact representation, TSD enables high-performance action recognition with a fraction of the data. Its lightweight nature and compatibility with existing 3D CNN backbones make it a versatile tool for efficient cloud-based video analysis.

Summary updated

Abstract

Video Analytics Software as a Service (VA SaaS) has been rapidly growing in recent years. VA SaaS is typically accessed by users using a lightweight client. Because the transmission bandwidth between the client and cloud is usually limited and expensive, it brings great benefits to design cloud video analysis algorithms with a limited data transmission requirement. Although considerable research has been devoted to video analysis, to our best knowledge, little of them has paid attention to the transmission bandwidth limitation in SaaS. As the first attempt in this direction, this work introduces a problem of few-frame action recognition, which aims at maintaining high recognition accuracy, when accessing only a few frames during both training and test. Unlike previous work that processed dense frames, we present Temporal Sequence Distillation (TSD), which distills a long video sequence into a very short one for transmission. By end-to-end training with 3D CNNs for video action recognition, TSD learns a compact and discriminative temporal and spatial representation of video frames. On Kinetics dataset, TSD+I3D typically requires only 50\% of the number of frames compared to I3D, a state-of-the-art video action recognition algorithm, to achieve almost the same accuracies. The proposed TSD has three appealing advantages. Firstly, TSD has a lightweight architecture and can be deployed in the client, eg. mobile devices, to produce compressed representative frames to save transmission bandwidth. Secondly, TSD significantly reduces the computations to run video action recognition with compressed frames on the cloud, while maintaining high recognition accuracies. Thirdly, TSD can be plugged in as a preprocessing module of any existing 3D CNNs. Extensive experiments show the effectiveness and characteristics of TSD.
BibTeX
@inproceedings{zhang2018temporal,
  author = {Zhaoyang Zhang and Zhanghui Kuang and Ping Luo and Litong Feng and Wei Zhang},
  title = {{Temporal Sequence Distillation: Towards Few-Frame Action Recognition in Videos}},
  booktitle = {ACM Multimedia},
  year = {2018},
  url = {https://openreview.net/forum?id=juOO9dITo-e},
}
Figure