Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency

Jia Li, Kui Fu, Shengwei Zhao, Shiming Ge

Abstract

The performance of video saliency estimation techniques has achieved significant advances along with the rapid development of Convolutional Neural Networks (CNNs). However, devices like cameras and drones may have limited computational capability and storage space so that the direct deployment of complex deep saliency models becomes infeasible. To address this problem, this paper proposes a dynamic saliency estimation approach for aerial videos via spatiotemporal knowledge distillation. In this approach, five components are involved, including two teachers, two students and the desired spatiotemporal model. The knowledge of spatial and temporal saliency is first separately transferred from the two complex and redundant teachers to their simple and compact students, and the input scenes are also degraded from high-resolution to low-resolution to remove the probable data redundancy so as to greatly speed up the feature extraction process. After that, the desired spatiotemporal model is further trained by distilling and encoding the spatial and temporal saliency knowledge of two students into a unified network. In this manner, the inter-model redundancy can be further removed for the effective estimation of dynamic saliency on aerial videos. Experimental results show that the proposed approach outperforms ten state-of-the-art models in estimating visual saliency on aerial videos, while its speed reaches up to 28,738 FPS on the GPU platform.

Introduction

Visual saliency estimation—predicting which parts of a video sequence will attract human attention—is a critical component for autonomous drone navigation and efficient video analysis. While deep learning has pushed the boundaries of accuracy, current state-of-the-art models are often too computationally expensive for deployment on edge devices. Drones, in particular, face severe resource constraints in terms of onboard memory and processing power.

Furthermore, high-resolution aerial footage often contains significant data redundancy. Much of the fine-grained detail in a high-definition frame is not strictly necessary to identify salient targets like moving vehicles or pedestrians. By recognizing that resolution itself can be a form of redundancy, we can significantly accelerate processing without sacrificing predictive power.

High-resolution vs. low-resolution saliency estimation for resource-constrained aerial platforms.

Methodology: The SKD Framework

We propose a two-step Spatiotemporal Knowledge Distillation (SKD) framework designed to produce a compact, high-speed saliency model. The core idea is to distill the expertise of complex "teacher" models into lightweight "student" models that operate on low-resolution inputs.

The proposed Spatiotemporal Knowledge Distillation (SKD) framework, featuring separate spatial/temporal distillation and joint spatiotemporal transfer.

1. Separate Spatial and Temporal Distillation

The process begins with two complex teacher models: one specialized in static spatial features and another in motion-based temporal features. These teachers process high-resolution video to generate high-quality saliency maps. Two lightweight student models are then trained to mimic these teachers. Crucially, the students are fed low-resolution versions of the same frames. This forces the students to learn the fundamental "essence" of saliency from a degraded signal, effectively removing data redundancy at the input level.

2. Joint Spatiotemporal Transfer

Once the spatial and temporal students are trained, their knowledge is fused into a single, unified Spatiotemporal Model ($S_{st}$). This second distillation step removes "inter-model redundancy"—the overlapping information that separate spatial and temporal streams might otherwise compute twice. The training utilizes a hybrid loss function that combines "soft labels" (the teachers' probabilistic predictions) with "hard labels" (actual human eye-tracking data from the AVS1K dataset).

Key Insights

  • Resolution Reduction: The study found that degrading input resolution (e.g., down to 32x32 or 64x64) has a negligible impact on saliency accuracy. This insight allows for a drastic reduction in the computational workload of feature extraction.
  • Probabilistic Learning: By using the teachers' soft labels, the student models gain a more nuanced understanding of the scene compared to learning from binary ground truth alone, which enhances generalization.
  • Modular Efficiency: Distilling spatial and temporal cues separately before fusion simplifies the optimization process, allowing a tiny network to effectively learn complex spatiotemporal features.

Results and Discussion

The proposed model was evaluated on the AVS1K dataset, the largest aerial video saliency dataset available, containing 1,000 videos and over 177,000 frames.

The SKD-DVA-64 variant achieved the following benchmarks:

  • Performance: Outperformed 10 state-of-the-art models (including DVA, SalNet, and STS) across metrics such as AUC, NSS, and Correlation Coefficient (CC).
  • Model Size: The model features only 0.30M parameters, a 98.8% reduction compared to the DVA teacher model.
  • Inference Speed:
    • GPU (NVIDIA 1080Ti): Reached an unprecedented 28,738 FPS (for the 32x32 version).
  • CPU (Intel 3.4GHz): Reached 1,490 FPS, demonstrating high viability for real-time processing on standard hardware.

Conclusion

The SKD framework provides a robust solution for real-time aerial saliency estimation. By combining resolution reduction with a multi-stage distillation process, the model achieves state-of-the-art accuracy while maintaining a minimal computational footprint. Future research will explore multi-teacher distillation and domain adaptation techniques to further broaden the model's applicability in diverse aerial surveillance and search-and-rescue missions.

Summary updated

Abstract

The performance of video saliency estimation techniques has achieved significant advances along with the rapid development of Convolutional Neural Networks (CNNs). However, devices like cameras and drones may have limited computational capability and storage space so that the direct deployment of complex deep saliency models becomes infeasible. To address this problem, this paper proposes a dynamic saliency estimation approach for aerial videos via spatiotemporal knowledge distillation. In this approach, five components are involved, including two teachers, two students and the desired spatiotemporal model. The knowledge of spatial and temporal saliency is first separately transferred from the two complex and redundant teachers to their simple and compact students, and the input scenes are also degraded from high-resolution to low-resolution to remove the probable data redundancy so as to greatly speed up the feature extraction process. After that, the desired spatiotemporal model is further trained by distilling and encoding the spatial and temporal saliency knowledge of two students into a unified network. In this manner, the inter-model redundancy can be further removed for the effective estimation of dynamic saliency on aerial videos. Experimental results show that the proposed approach outperforms ten state-of-the-art models in estimating visual saliency on aerial videos, while its speed reaches up to 28,738 FPS on the GPU platform.
BibTeX
@article{li2020spatiotemporal,
  author = {Jia Li and Kui Fu and Shengwei Zhao and Shiming Ge},
  title = {{Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency}},
  journal = {IEEE Transactions on Image Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=e5XFMuORux},
}
Figure