Event and Activity Recognition in Video Surveillance for Cyber-Physical Systems

Swarnabja Bhaumik, Prithwish Jana, Partha Pratim MohantaarXiv 2021 · DOI · OpenReview

Introduction

In the evolving landscape of Cyber-Physical Systems (CPS), the automated understanding of events and activities within video surveillance is a critical capability. However, real-world footage is often "unconstrained," captured by drones, CCTVs, or low-end mobile devices. These videos frequently suffer from poor lighting, camera jitter, and cluttered backgrounds. Historically, approaches to this problem have ranged from early Structure from Motion (SFM) techniques to modern deep learning frameworks. This paper presents a robust hybrid CNN-RNN architecture designed to recognize complex events in these difficult environments by emphasizing long-term motion patterns and intelligent data fusion.

Methodology

The proposed system employs a multi-stage pipeline focused on computational efficiency and intelligent data fusion to handle the complexities of unconstrained video.

Graph-Based Key-Frame Sampling

Processing every frame in a high-resolution video is computationally expensive and often redundant. To address this, the framework uses a two-stage reduction process. Initially, frames with similar motion histograms are pruned. Subsequently, the remaining frames are modeled as nodes in a graph. A "viability" algorithm then selects a fixed number of distinctive key-frames that are both visually unique and temporally representative of the entire sequence.

A video sequence of a horse race, representing the unconstrained surveillance footage processed by the system.

Hybrid CNN-RNN Architecture

The architecture consists of two primary streams:

  • Spatial Stream: A ResNet-50 Convolutional Neural Network (CNN) extracts features from raw RGB key-frames.
  • Temporal Stream: The same architecture processes optical-flow magnitude images to capture long-term motion patterns.

These extracted features are then fed into a Long Short-Term Memory (LSTM) network. This Recurrent Neural Network (RNN) component is vital for modeling the temporal dependencies and the sequential order of events across the selected key-frames.

Multi-Tier Biased Conflation

The core innovation lies in the fusion strategy. Rather than simple late fusion (averaging prediction scores), the authors introduce "multi-tier biased conflation." This technique utilizes the Bhattacharyya distance to weight predictions toward the more confident distribution. This allows the model to achieve a powerful consensus, ensuring that a clear motion signal in the temporal stream isn't drowned out by a misleading spatial background, or vice versa.

Experimental Results

The framework was rigorously evaluated on four benchmark datasets: Columbia Consumer Videos (CCV), Kodak Consumer Video (KCV), UCF-101, and Human Motion Database (HMDB-51).

The results demonstrate significant improvements over state-of-the-art (SoTA) methods:

  • CCV Dataset: The model achieved an accuracy of 81.89%, representing a 15.31% improvement over the best individual modality.
  • Temporal Performance: Notably, the temporal stream alone achieved 79.13% on CCV, outperforming many existing fused spatio-temporal models.
  • Scalability: On the UCF-101 dataset, the fused model reached 89.03% accuracy, proving its effectiveness across a large number of action classes.

Conclusion and Future Work

The research highlights that focusing on long-term motion patterns and sophisticated fusion strategies leads to more robust event recognition in unconstrained environments. By reducing video data to sparse, distinctive key-frames, the system maintains high accuracy while remaining efficient enough for real-time CPS applications.

Acoustic and textual modalities identified as potential areas for future multi-modal integration.

Future work will explore the integration of additional modalities, such as acoustic signals and scene text, to further enhance recognition in noisy environments. Additionally, the incorporation of attention mechanisms could help the model better isolate semantically important regions within cluttered frames.

Summary updated

Abstract

This chapter aims to aid the development of Cyber-Physical Systems (CPS) in automated understanding of events and activities in various applications of video-surveillance. These events are mostly captured by drones, CCTVs or novice and unskilled individuals on low-end devices. Being unconstrained, these videos are immensely challenging due to a number of quality factors. We present an extensive account of the various approaches taken to solve the problem over the years. This ranges from methods as early as Structure from Motion (SFM) based approaches to recent solution frameworks involving deep neural networks. We show that the long-term motion patterns alone play a pivotal role in the task of recognizing an event. Consequently each video is significantly represented by a fixed number of key-frames using a graph-based approach. Only the temporal features are exploited using a hybrid Convolutional Neural Network (CNN) + Recurrent Neural Network (RNN) architecture. The results we obtain are encouraging as they outperform standard temporal CNNs and are at par with those using spatial information along with motion cues. Further exploring multistream models, we conceive a multi-tier fusion strategy for the spatial and temporal wings of a network. A consolidated representation of the respective individual prediction vectors on video and frame levels is obtained using a biased conflation technique. The fusion strategy endows us with greater rise in precision on each stage as compared to the state-of-the-art methods, and thus a powerful consensus is achieved in classification. Results are recorded on four benchmark datasets widely used in the domain of action recognition, namely CCV, HMDB, UCF-101 and KCV. It is inferable that focusing on better classification of the video sequences certainly leads to robust actuation of a system designed for event surveillance and object cum activity tracking.
BibTeX
@misc{bhaumik2021event,
  author = {Swarnabja Bhaumik and Prithwish Jana and Partha Pratim Mohanta},
  title = {{Event and Activity Recognition in Video Surveillance for Cyber-Physical Systems}},
  year = {2021},
  eprint = {2111.02064},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.02064},
}
Figure