Event and Activity Recognition in Video Surveillance for Cyber-Physical Systems

Swarnabja Bhaumik, Prithwish Jana, Partha Pratim MohantaAdvances 2021 · DOI · arXiv · OpenAlex

Abstract

This paper introduces a robust framework for recognizing complex events and activities in unconstrained surveillance videos, such as those from drones or CCTV. By integrating a graph-based key-frame selection process, a hybrid CNN-RNN architecture, and a novel "multi-tier biased conflation" fusion strategy, the proposed method achieves state-of-the-art performance across several major benchmarks.

Introduction

Automated video surveillance in Cyber-Physical Systems (CPS) faces significant hurdles due to "unconstrained" video conditions. Unlike curated datasets, real-world surveillance footage is often plagued by camera jitter, poor lighting, cluttered backgrounds, and massive temporal redundancy. Traditional methods frequently struggle to balance spatial details (what is in the frame) with temporal dynamics (how things move over time). Furthermore, simple fusion techniques like averaging predictions often fail to account for the varying confidence levels of different models.

Methodology

The proposed pipeline transforms raw video into accurate event classifications through three primary stages:

1. Smart Key-Frame Sampling

To handle redundancy, a two-stage process is employed. First, "motion-redundant" frames are discarded using optical flow and motion histograms. Second, a graph-based approach selects the most "distinctive" frames, ensuring they are sufficiently spaced in time to capture the full semantic narrative of the video.

2. Hybrid CNN-RNN Architecture

The framework utilizes a dual-stream approach to process visual information:

  • Spatial Stream (CNN): A ResNet-50 model extracts features from individual RGB key-frames to capture static scene details.
  • Temporal Stream (CNN): The same architecture processes optical flow magnitude images to capture motion patterns.
  • Sequence Modeling (RNN): Long Short-Term Memory (LSTM) networks take the sequence of frame-level features to learn long-term temporal patterns for video-level prediction.
The spatial stream processes RGB images to extract scene-level features.
The temporal stream utilizes optical flow to capture motion dynamics within the video.

3. Multi-Tier Biased Conflation

The core novelty lies in the fusion strategy. Instead of simple averaging, the authors use "biased conflation" to merge probability distributions. This statistical technique uses the Bhattacharyya distance to bias the final result toward the more certain or representative distribution, effectively reconciling frame-level and video-level predictions across spatial and temporal modalities.

Experimental Evaluation

The framework was evaluated on four widely recognized benchmarks: Columbia Consumer Videos (CCV), Kodak Consumer Videos (KCV), UCF-101, and the Human Motion Database (HMDB-51).

Key Results

The multi-tier fusion strategy demonstrated significant accuracy improvements over individual modality streams:

  • CCV: Achieved 81.89% accuracy (a 15.31% boost over the best single stream).
  • UCF-101: Achieved 89.03% accuracy.
  • HMDB-51: Achieved 61.91% accuracy.
  • KCV: Achieved 57.52% accuracy.

In almost all cases, the biased conflation method outperformed standard state-of-the-art fusion techniques like Borda-count or simple averaging.

Discussion and Insights

Analysis of the results suggests that long-term motion patterns are often more pivotal for event recognition than spatial details alone; the standalone temporal stream outperformed several existing spatio-temporal models on the CCV and KCV datasets. Additionally, the study confirms that standard "late fusion" can create a sub-optimal consensus, whereas biased conflation provides a more robust statistical reconciliation against outliers or weak predictions.

Conclusion and Future Work

While the current framework focuses on spatial and temporal cues, future work could integrate acoustic and textual channels for more comprehensive recognition. Additionally, while key-frame sampling improves efficiency, the calculation of dense optical flow remains computationally intensive. Future directions include exploring attention mechanisms or "fovea" streams to refine the model's focus in crowded or complex scenes.

Summary updated

BibTeX
@inproceedings{bhaumik2021event,
  author = {Swarnabja Bhaumik and Prithwish Jana and Partha Pratim Mohanta},
  title = {{Event and Activity Recognition in Video Surveillance for Cyber-Physical Systems}},
  booktitle = {Advances},
  year = {2021},
  doi = {10.1007/978-3-030-66222-6\_4},
}
Figure