Loss Switching Fusion with Similarity Search for Video Classification

Lei Wang, Du Q. Huynh, Moussa Reda Mansour

Abstract

Video classification in outdoor environments remains a significant challenge due to dynamic background motions such as swaying trees and weather conditions. This paper introduces LSFNet (Loss Switching Fusion Network), a lightweight system that integrates traditional handcrafted features with a specialized neural network architecture. By employing an alternating training strategy between reconstruction and classification losses, LSFNet learns robust feature representations. Experimental results on industrial surveillance datasets demonstrate that LSFNet outperforms heavy deep learning models like C3D and I3D, achieving over 90% accuracy in foreground motion detection while maintaining computational efficiency.

1. Introduction

Outdoor video classification is complicated by "dynamic background motions"—environmental factors like rain, illumination changes, and camera shake—that often mimic foreground human activities. Standard deep learning models (CNNs) typically require massive labeled datasets and high computational power, making them difficult to deploy in specific industrial contexts. This work proposes a hybrid approach that leverages spatiotemporal features and a novel loss-switching mechanism to create a unified, efficient representation for both background and foreground classification.

2. Methodology

2.1 Spatiotemporal Feature Extraction

The system utilizes "Improved Trajectories" (iDT) as the primary input. These handcrafted descriptors capture motion through Spatio-Temporal Histograms of Optical Flow (ST-HOF), Motion Boundary Histograms (ST-MBH), and Trajectory descriptors (ST-DT), alongside appearance via Histograms of Oriented Gradients (ST-HOG).

2.2 LSFNet Architecture and Loss Switching

The core of the system is the Loss Switching Fusion Network (LSFNet), a 5-layer autoencoder integrated with a Multi-Layer Perceptron (MLP).

Figure 1: The LSFNet architecture, showing the fusion of spatiotemporal features and the dual-loss training mechanism.

The innovation lies in the training phase, where the network alternates between two loss functions:

  • Mean Squared Error (MSE) Loss: Forces the network to learn how to reconstruct the input, preserving general structural information.
  • Cross-Entropy Loss: Forces the network to make the features more discriminative for specific classes.

This dual-steering approach produces low-dimensional features that are both representative and highly separable.

2.3 Feature Selection and Classification

To further refine the representation, Fisher Scores are used to select the top 50% of feature components. For final classification, the system employs Locality Sensitive Hashing (LSH) for similarity search. Instead of a standard softmax output, the system identifies the "nearest neighbors" in the training set and uses soft voting to assign labels, enhancing robustness in noisy environments.

3. Experimental Results

3.1 Datasets

Evaluation was conducted on two private industrial datasets from iCetana, containing over 9,000 videos from outdoor environments like car parks and train stations. Labels included six background classes (e.g., tree waving, rainy) and three foreground human motion classes.

3.2 Performance Comparison

LSFNet significantly outperformed fine-tuned state-of-the-art models. As shown in the table below, the system achieved a substantial lead in both background and foreground classification tasks.

Algorithm Background Env. Motion Foreground Human Motion
C3D (Fine-tuned) 75.9% 77.8%
I3D RGB (Fine-tuned) 77.0% 79.9%
LSFNet (Whole System) 88.9% 90.7%

3.3 Feature Visualization

The effectiveness of the loss-switching strategy is qualitatively demonstrated through UMAP visualizations.

Figure 5: UMAP visualizations comparing feature embeddings from simple concatenation, a standard autoencoder, and the proposed LSFNet fusion.

As seen in Figure 5, LSFNet produces significantly tighter and more distinct clusters for both background types (e) and human motion detection (f) compared to baseline methods like PCA or standalone autoencoders.

4. Conclusion

LSFNet demonstrates that combining handcrafted features with a targeted neural fusion strategy can surpass heavy-duty deep learning models in specialized, noisy domains. By using a single feature representation for multiple tasks, the system offers a computationally efficient solution for real-time industrial surveillance. Future work will explore integrating higher-level semantic features into the fusion pipeline.

Summary updated

Abstract

From video streaming to security and surveillance applications, video data play an important role in our daily living today. However, managing a large amount of video data and retrieving the most useful information for the user remain a challenging task. In this paper, we propose a novel video classification system that would benefit the scene understanding task. We define our classification problem as classifying background and foreground motions using the same feature representation for outdoor scenes. This means that the feature representation needs to be robust enough and adaptable to different classification tasks. We propose a lightweight Loss Switching Fusion Network (LSFNet) for the fusion of spatiotemporal descriptors and a similarity search scheme with soft voting to boost the classification performance. The proposed system has a variety of potential applications such as content-based video clustering, video filtering, etc. Evaluation results on two private industry datasets show that our system is robust in both classifying different background motions and detecting human motions from these background motions.
BibTeX
@inproceedings{wang2019loss,
  author = {Lei Wang and Du Q. Huynh and Moussa Reda Mansour},
  title = {{Loss Switching Fusion with Similarity Search for Video Classification}},
  booktitle = {ICIP},
  year = {2019},
  url = {https://openreview.net/forum?id=xc0XO_KsEA},
}
Figure