Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency

Jia Li, Kui Fu, Shengwei Zhao, Shiming Ge

TL;DR

This paper introduces a two-step Spatiotemporal Knowledge Distillation (SKD) framework designed to estimate visual saliency in aerial videos at lightning speeds. By distilling knowledge from complex "teacher" models into compact "students" while simultaneously slashing input resolution, the authors achieved state-of-the-art accuracy with a staggering inference speed of up to 28,738 FPS on a modern GPU.

What Problem Does This Paper Address?

Visual saliency estimation—predicting which parts of a video will grab a human viewer's attention—is crucial for autonomous drone navigation and efficient video analysis. However, current deep learning models are "ground-level" heavyweights: they are designed for high-resolution desktop processing and struggle with the unique viewpoints of aerial footage.

Drones face two major hurdles:

  1. Resource Constraints: Limited onboard memory and computational power make deploying massive CNNs impossible.
  2. Data Redundancy: High-resolution aerial frames contain massive amounts of detail that aren't actually necessary to identify salient targets (like a lone hiker or a moving vehicle).

Methods

The authors propose a "teacher-student" architecture that removes redundancy in two distinct phases:

  1. Separate Spatial and Temporal Distillation:
    • Two complex "Teacher" models (one for static spatial features, one for motion/temporal features) process high-resolution video.
  2. Two lightweight "Student" models are trained to mimic these teachers but using low-resolution inputs. This forces the students to learn the "essence" of saliency without needing fine-grained pixel details.
  3. Joint Spatiotemporal Transfer:
    • The knowledge from both the spatial and temporal students is fused into a single, unified Spatiotemporal Model ($S_{st}$).
  4. This step removes "inter-model redundancy"—the overlapping information that both spatial and temporal streams might otherwise compute twice.

The training uses a hybrid loss function: a soft label loss (learning from the teachers' probabilistic outputs) and a hard label loss (learning from actual human eye-tracking data).

Datasets

The models were evaluated on AVS1K (Aerial Video Saliency 1K), currently the largest dataset of its kind. It contains:

  • 1,000 aerial videos (177,644 frames).
  • Four categories: Building, Human, Vehicle, and Others.
  • Ground-truth "fixation density maps" derived from human eye-tracking data.

Key Insights and Ideas

  • Resolution as Redundancy: One of the most counter-intuitive findings is that degrading resolution (e.g., down to 32x32 or 64x64) barely hurts saliency accuracy. Humans can easily spot a salient object in a blurry thumbnail; the authors proved that compact neural networks can do the same, drastically reducing the "feature extraction" workload.
  • The Power of "Soft Labels": By using the teachers' soft predictions rather than just binary ground truth, the student models gain a better "probabilistic understanding" of the scene, which improves their ability to generalize to new, unseen aerial environments.
  • Step-wise Efficiency: By distilling spatial and temporal cues separately before fusing them, the model avoids the optimization difficulties of trying to learn complex 3D (spatiotemporal) features from scratch in a tiny network.

Key Results

The proposed SKD-DVA-64 model (using a 64x64 input) achieved impressive benchmarks:

  • Accuracy: Outperformed 10 state-of-the-art models (including DVA, SalNet, and STS) across metrics like AUC, NSS, and Correlation Coefficient (CC).
  • Efficiency: The model has only 0.30M parameters, a 98.8% reduction compared to the DVA teacher model.
  • Speed:
    • GPU (NVIDIA 1080Ti): Reached 28,738 FPS (for the 32x32 version).
  • CPU (Intel 3.4GHz): Reached 1,490 FPS, making it highly viable for real-time processing on low-power hardware.

Limitations and Future Work

While the results are robust, the authors note a few areas for growth:

  • Multi-Teacher Distillation: Currently, they distill from one spatial and one temporal teacher separately. They suggest exploring simultaneous distillation from multiple diverse models to capture even richer feature sets.
  • Domain Adaptation: They aim to explore how to better "speed up" the transfer of knowledge from one specific domain (e.g., urban surveillance) to another (e.g., wildland search and rescue) without retraining from scratch.
  • Data Distillation: Beyond model and resolution reduction, they see potential in distilling the temporal data itself to further accelerate processing.

Summary updated

Abstract

The performance of video saliency estimation techniques has achieved significant advances along with the rapid development of Convolutional Neural Networks (CNNs). However, devices like cameras and drones may have limited computational capability and storage space so that the direct deployment of complex deep saliency models becomes infeasible. To address this problem, this paper proposes a dynamic saliency estimation approach for aerial videos via spatiotemporal knowledge distillation. In this approach, five components are involved, including two teachers, two students and the desired spatiotemporal model. The knowledge of spatial and temporal saliency is first separately transferred from the two complex and redundant teachers to their simple and compact students, while the input scenes are also degraded from high-resolution to low-resolution to remove the probable data redundancy so as to greatly speed up the feature extraction process. After that, the desired spatiotemporal model is further trained by distilling and encoding the spatial and temporal saliency knowledge of two students into a unified network. In this manner, the inter-model redundancy can be removed for the effective estimation of dynamic saliency on aerial videos. Experimental results show that the proposed approach is comparable to 11 state-of-the-art models in estimating visual saliency on aerial videos, while its speed reaches up to 28,738 FPS and 1,490.5 FPS on the GPU and CPU platforms, respectively.
BibTeX
@article{li2020spatiotemporal,
  author = {Jia Li and Kui Fu and Shengwei Zhao and Shiming Ge},
  title = {{Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency}},
  journal = {IEEE Transactions on Image Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=e5XFMuORux},
}

Loading related content...

Figure