Abstract
Video Analytics Software as a Service (VA SaaS) is rapidly expanding, typically involving lightweight clients transmitting data to cloud-based servers. However, limited and expensive transmission bandwidth creates a significant bottleneck. This paper introduces Temporal Sequence Distillation (TSD), a novel approach for few-frame action recognition. TSD distills long video sequences into very short, representative ones for transmission. By training end-to-end with 3D CNNs, TSD learns compact temporal and spatial representations. On the Kinetics dataset, the TSD+I3D framework achieves comparable accuracy to state-of-the-art models while requiring only 50% of the frames, significantly reducing both bandwidth and cloud computation costs.
The Challenge: Bandwidth in Video Analytics SaaS
The growth of VA SaaS faces a major hurdle: the "client-cloud" gap. Lightweight clients, such as mobile devices or smart cameras, must send video data to powerful cloud servers for complex tasks like action recognition. While 3D Convolutional Neural Networks (3D CNNs) are highly effective, they typically require dense frame stacks (often 64 or more) to capture sufficient temporal context. Transmitting such high-resolution, high-frame-rate video is bandwidth-intensive and costly.
To bridge this gap, we propose few-frame action recognition, aiming to maintain high recognition accuracy while accessing only a minimal number of frames during both training and testing.
Temporal Sequence Distillation (TSD)
TSD is a lightweight "plug-and-play" module designed to be deployed on the client side. Instead of simple random or uniform sampling, TSD learns to combine a long sequence into a very short, "distilled" one.
The architecture consists of a coarse feature extractor (MobileNet) and a TSD block that calculates a transformation matrix. The resulting distilled frames are linear combinations of the input frames, effectively acting as a "weighted average" optimized for recognition performance. This process often results in a "ghosting" or motion-blur effect in the distilled frames, which serves to encode motion information directly into the static representation.
Key Results and Performance
Extensive experiments on the Kinetics and UCF101 datasets demonstrate that TSD significantly improves efficiency without sacrificing accuracy. TSD identifies and merges the most informative parts of a video, eliminating the redundancy inherent in high-frame-rate clips.
Key findings include:
- Efficiency Gains: On Kinetics, I3D+TSD achieved 70.7% accuracy using only 60 frames, whereas the baseline I3D required 250 frames to reach a similar 71.1%.
- Low-Frame Superiority: When restricted to a very small budget (e.g., 5 frames), TSD improved I3D's accuracy on Kinetics by 29.2% compared to standard uniform sampling.
- Cloud Speedup: By reducing the frame count, TSD cut cloud-side computation (FLOPs) by approximately 50% and increased processing speeds by 45%–48%.
Conclusion
Temporal Sequence Distillation offers a practical solution for the bandwidth constraints of modern VA SaaS. By learning to distill motion and spatial features into a compact representation, TSD enables high-performance action recognition with a fraction of the data. Its lightweight nature and compatibility with existing 3D CNN backbones make it a versatile tool for efficient cloud-based video analysis.
Summary updated