Saliency-guided video classification via adaptively weighted learning

Yunzhen Zhao, Yuxin Peng

Abstract

Video classification is productive in many practical applications, and the recent deep learning has greatly improved its accuracy. However, existing works often model video frames indiscriminately, but from the view of motion, video frames can be decomposed into salient and non-salient areas naturally. Salient and non-salient areas should be modeled with different networks, for the former present both appearance and motion information, and the latter present static background information. To address this problem, in this paper, video saliency is predicted by optical flow without supervision firstly. Then two streams of 3D CNN are trained individually for raw frames and optical flow on salient areas, and another 2D CNN is trained for raw frames on non-salient areas. For the reason that these three streams play different roles for each class, the weights of each stream are adaptively learned for each class. Experimental results show that saliency-guided modeling and adaptively weighted learning can reinforce each other, and we achieve the state-of-the-art results.

Introduction

Standard video classification models often treat every pixel in a frame with equal importance. However, a video is naturally composed of two distinct parts: the salient areas (where the action happens) and the non-salient areas (the static background).

The authors argue that these regions provide different types of information. Salient areas contain both appearance and motion data, while non-salient areas primarily provide static context. Existing methods that model the entire frame indiscriminately often struggle to distinguish between object motion and camera motion, or they fail to leverage the specific contextual clues hidden in the background. The challenge lies in automatically identifying these regions without manual labeling and determining how much "weight" to give each region depending on the type of video being classified.

Methodology

The proposed framework consists of three main stages: unsupervised saliency prediction, a three-stream architecture, and adaptively weighted learning.

Overview of the proposed three-stream saliency-guided video classification framework.

Unsupervised Saliency Prediction

To identify action regions without human labels, the authors utilize optical flow. They first estimate and eliminate camera motion (using homography and RANSAC) to ensure they are tracking only moving objects. Edge detection is then applied to the resulting flow field to define rectangular salient regions.

Three-Stream Architecture

The model processes the decomposed video through three specialized networks:

  • Stream 1 (Salient Raw): A 3D CNN (based on the C3D architecture) processes raw frames of salient areas to capture appearance and local motion.
  • Stream 2 (Salient Flow): Another 3D CNN processes the optical flow of salient areas to focus purely on motion patterns.
  • Stream 3 (Non-salient Raw): A 2D CNN (VGG-19) processes the background areas. Since the background is mostly static, a 2D network is more efficient and sufficient for capturing environmental context.

Adaptively Weighted Learning

Instead of simple averaging, the model learns a unique set of weights for every class. For example, a "Basketball" video might rely heavily on motion in the salient stream, while a "Birthday Party" video might rely more on the background appearance. These weights are optimized using linear programming.

Experimental Results

The framework was evaluated on two widely used benchmarks: UCF-101 (13,320 YouTube clips across 101 action categories) and Columbia Consumer Videos (CCV) (9,317 web videos across 20 semantic categories).

Comparison of the proposed method against state-of-the-art techniques on UCF-101 and CCV datasets.

The framework achieved state-of-the-art performance at the time of publication:

  • UCF-101: Reached 94.4% accuracy, significantly outperforming the standard C3D model (90.4%) and other multi-stream approaches.
  • CCV: Reached 85.9% mAP (mean Average Precision).
  • Ablation Study: The authors showed that modeling saliency specifically provided a consistent boost of roughly 1.5% to 2.5% across different fusion methods compared to modeling the whole frame.

Discussion and Conclusion

This research highlights that background context is a signal, not noise. Using a dedicated 2D CNN for the background allows the model to "understand" the setting without wasting the computational overhead of a 3D CNN on static pixels. Furthermore, class-specific fusion allows the model to tune itself for each specific category based on whether it is defined by motion or environment.

While the current method relies on optical flow for saliency—which may struggle in complex lighting or overlapping motion—it proves that effectively "cropping" a video to its most important parts without manual labels is highly effective. Future work may explore more advanced saliency models or end-to-end trainable fusion layers to further optimize the system.

Summary updated

Abstract

Video classification is productive in many practical applications, and the recent deep learning has greatly improved its accuracy. However, existing works often model video frames indiscriminately, but from the view of motion, video frames can be decomposed into salient and non-salient areas naturally. Salient and non-salient areas should be modeled with different networks, for the former present both appearance and motion information, and the latter present static background information. To address this problem, in this paper, video saliency is predicted by optical flow without supervision firstly. Then two streams of 3D CNN are trained individually for raw frames and optical flow on salient areas, and another 2D CNN is trained for raw frames on non-salient areas. For the reason that these three streams play different roles for each class, the weights of each stream are adaptively learned for each class. Experimental results show that saliency-guided modeling and adaptively weighted learning can reinforce each other, and we achieve the state-of-the-art results.
BibTeX
@inproceedings{zhao2017saliency,
  author = {Yunzhen Zhao and Yuxin Peng},
  title = {{Saliency-guided video classification via adaptively weighted learning}},
  booktitle = {ICME},
  year = {2017},
  doi = {10.1109/icme.2017.8019343},
}
Figure