Abstract
Video classification is productive in many practical applications, and the recent deep learning has greatly improved its accuracy. However, existing works often model video frames indiscriminately, but from the view of motion, video frames can be decomposed into salient and non-salient areas naturally. Salient and non-salient areas should be modeled with different networks, for the former present both appearance and motion information, and the latter present static background information. To address this problem, in this paper, video saliency is predicted by optical flow without supervision firstly. Then two streams of 3D CNN are trained individually for raw frames and optical flow on salient areas, and another 2D CNN is trained for raw frames on non-salient areas. For the reason that these three streams play different roles for each class, the weights of each stream are adaptively learned for each class. Experimental results show that saliency-guided modeling and adaptively weighted learning can reinforce each other, and we achieve the state-of-the-art results.
Introduction
Standard video classification models often treat every pixel in a frame with equal importance. However, a video is naturally composed of two distinct parts: the salient areas (where the action happens) and the non-salient areas (the static background).
The authors argue that these regions provide different types of information. Salient areas contain both appearance and motion data, while non-salient areas primarily provide static context. Existing methods that model the entire frame indiscriminately often struggle to distinguish between object motion and camera motion, or they fail to leverage the specific contextual clues hidden in the background. The challenge lies in automatically identifying these regions without manual labeling and determining how much "weight" to give each region depending on the type of video being classified.
Methodology
The proposed framework consists of three main stages: unsupervised saliency prediction, a three-stream architecture, and adaptively weighted learning.
Unsupervised Saliency Prediction
To identify action regions without human labels, the authors utilize optical flow. They first estimate and eliminate camera motion (using homography and RANSAC) to ensure they are tracking only moving objects. Edge detection is then applied to the resulting flow field to define rectangular salient regions.
Three-Stream Architecture
The model processes the decomposed video through three specialized networks:
- Stream 1 (Salient Raw): A 3D CNN (based on the C3D architecture) processes raw frames of salient areas to capture appearance and local motion.
- Stream 2 (Salient Flow): Another 3D CNN processes the optical flow of salient areas to focus purely on motion patterns.
- Stream 3 (Non-salient Raw): A 2D CNN (VGG-19) processes the background areas. Since the background is mostly static, a 2D network is more efficient and sufficient for capturing environmental context.
Adaptively Weighted Learning
Instead of simple averaging, the model learns a unique set of weights for every class. For example, a "Basketball" video might rely heavily on motion in the salient stream, while a "Birthday Party" video might rely more on the background appearance. These weights are optimized using linear programming.
Experimental Results
The framework was evaluated on two widely used benchmarks: UCF-101 (13,320 YouTube clips across 101 action categories) and Columbia Consumer Videos (CCV) (9,317 web videos across 20 semantic categories).
The framework achieved state-of-the-art performance at the time of publication:
- UCF-101: Reached 94.4% accuracy, significantly outperforming the standard C3D model (90.4%) and other multi-stream approaches.
- CCV: Reached 85.9% mAP (mean Average Precision).
- Ablation Study: The authors showed that modeling saliency specifically provided a consistent boost of roughly 1.5% to 2.5% across different fusion methods compared to modeling the whole frame.
Discussion and Conclusion
This research highlights that background context is a signal, not noise. Using a dedicated 2D CNN for the background allows the model to "understand" the setting without wasting the computational overhead of a 3D CNN on static pixels. Furthermore, class-specific fusion allows the model to tune itself for each specific category based on whether it is defined by motion or environment.
While the current method relies on optical flow for saliency—which may struggle in complex lighting or overlapping motion—it proves that effectively "cropping" a video to its most important parts without manual labels is highly effective. Future work may explore more advanced saliency models or end-to-end trainable fusion layers to further optimize the system.
Summary updated