Temporal Sequence Distillation: Towards Few-Frame Action Recognition in Videos

Zhaoyang Zhang, Zhanghui Kuang, Ping Luo, Litong Feng, Wei Zhang

TL;DR

Temporal Sequence Distillation (TSD) is a lightweight "plug-and-play" module that compresses long video sequences into just a few informative frames. By learning to distill motion and spatial features into a compact representation, TSD allows 3D CNNs to achieve state-of-the-art action recognition accuracy while using 50% fewer frames, significantly reducing data transmission and cloud computation costs.

What Problem Does This Paper Address?

Video Analytics Software as a Service (VA SaaS) is booming, but it faces a major bottleneck: the "client-cloud" gap. In a typical setup, a lightweight client (like a mobile phone or a smart camera) sends video data to a powerful cloud server for analysis (e.g., recognizing an action).

However, transmitting high-resolution, high-frame-rate video is expensive and consumes massive bandwidth. While 3D Convolutional Neural Networks (3D CNNs) are the gold standard for action recognition, they typically require dense stacks of frames (often 64 or more) to capture temporal context. This creates a conflict: the cloud needs many frames for accuracy, but the client can only afford to send a few. The authors propose few-frame action recognition to bridge this gap, aiming for high accuracy with minimal data transmission.

Methods

The core innovation is Temporal Sequence Distillation (TSD). Instead of just randomly or uniformly sampling frames (which might miss the "action"), TSD learns how to combine a long sequence into a very short, "distilled" one.

  1. The Architecture: TSD consists of three parts:
  2. Coarse Feature Extractor: A lightweight MobileNet that extracts basic features from the raw video snippets.
  3. TSD Block: This is the "brain" of the operation. It uses a dual-path network (temporal and feature embedding) to calculate a transformation matrix $P$.
  4. Distillation Formula: The distilled sequence $Y$ is a linear combination of the input frames $X$, defined as $Y = XP$. Essentially, each distilled frame is a "weighted average" of the original frames, where the weights are learned to maximize recognition performance.
  5. The "Ghosting" Effect: Because distilled frames are combinations of multiple original frames, they often exhibit a "ghosting" or motion-blur appearance. This isn't a bug; it’s a feature. These frames effectively encode motion information directly into the static image.
  6. Deployment: TSD is designed for the SaaS model. The lightweight feature extractor and TSD block run on the client (saving bandwidth by sending fewer frames), while the heavy-duty recognition network (like I3D) runs on the cloud.

Datasets

The authors evaluated TSD on two benchmark datasets for video action recognition:

  • Kinetics: A large-scale dataset containing 240,000 training videos across 400 human action classes.
  • UCF101: A classic dataset with 13,320 videos from 101 action categories.

Key Insights and Ideas

  • Redundancy is the Enemy: Videos are incredibly redundant. Most frames in a 10-second clip don't actually contribute to identifying the action. TSD identifies and merges the "important" parts.
  • Learned Sampling > Heuristic Sampling: Traditional methods use uniform or random sampling. TSD proves that learning how to sample (and combine) frames provides a massive accuracy boost when the frame budget is low.
  • Efficiency without Sacrifice: On the Kinetics dataset, TSD + I3D achieved nearly the same accuracy as a standard I3D model while requiring only half the number of frames.
  • End-to-End Synergy: TSD isn't a separate preprocessing step; it is trained end-to-end with the main recognition network. This ensures the distilled frames are specifically optimized for what the 3D CNN needs to see.

Key Results

  • Accuracy vs. Efficiency: On Kinetics, I3D+TSD achieved 70.7% accuracy using only 60 frames. For comparison, the baseline I3D required 250 frames to reach a similar 71.1%.
  • Cloud Speedup: By reducing the number of frames the cloud has to process, TSD reduced cloud-side computation (FLOPs) by roughly 50% and increased processing speed by 45%–48%.
  • Performance Gains: When restricted to very few frames (e.g., only 5 frames), TSD improved I3D's accuracy on Kinetics by a staggering 29.2% compared to standard sampling.

Limitations and Future Work

  • Linear Assumption: The current model assumes distilled frames are a linear combination of input frames ($Y=XP$). The authors suggest that non-linear distillation methods might yield even richer representations.
  • Client-Side Overhead: While TSD saves bandwidth and cloud compute, it does add a small computational load to the client device (the MobileNet extractor). For extremely low-power IoT devices, even this might be a consideration.
  • Generalization: While tested with I3D, the authors note that TSD can be plugged into any 3D CNN, but more extensive testing across different backbones (like SlowFast or X3D) would further validate its universality.

Summary updated

Abstract

Video Analytics Software as a Service (VA SaaS) has been rapidly growing in recent years. VA SaaS is typically accessed by users using a lightweight client. Because the transmission bandwidth between the client and cloud is usually limited and expensive, it brings great benefits to design cloud video analysis algorithms with a limited data transmission requirement. Although considerable research has been devoted to video analysis, to our best knowledge, little of them has paid attention to the transmission bandwidth limitation in SaaS. As the first attempt in this direction, this work introduces a problem of few-frame action recognition, which aims at maintaining high recognition accuracy, when accessing only a few frames during both training and test. Unlike previous work that processed dense frames, we present Temporal Sequence Distillation (TSD), which distills a long video sequence into a very short one for transmission. By end-to-end training with 3D CNNs for video action recognition, TSD learns a compact and discriminative temporal and spatial representation of video frames. On Kinetics dataset, TSD+I3D typically requires only 50% of the number of frames compared to I3D, a state-of-the-art video action recognition algorithm, to achieve almost the same accuracies. The proposed TSD has three appealing advantages. Firstly, TSD has a lightweight architecture and can be deployed in the client, eg., mobile devices, to produce compressed representative frames to save transmission bandwidth. Secondly, TSD significantly reduces the computations to run video action recognition with compressed frames on the cloud, while maintaining high recognition accuracies. Thirdly, TSD can be plugged in as a preprocessing module of any existing 3D CNNs. Extensive experiments show the effectiveness and characteristics of TSD.
BibTeX
@inproceedings{zhang2018temporal,
  author = {Zhaoyang Zhang and Zhanghui Kuang and Ping Luo and Litong Feng and Wei Zhang},
  title = {{Temporal Sequence Distillation: Towards Few-Frame Action Recognition in Videos}},
  booktitle = {ACM Multimedia},
  year = {2018},
  url = {https://openreview.net/forum?id=juOO9dITo-e},
}

Loading related content...

Figure