AVS-Net: Automatic Visual Surveillance Using Relation Network

Sein Jang, Young-Ho Park, Aziz NasridinovAAAI 2019 · DOI

Introduction

Automated visual surveillance is a critical component of modern public safety infrastructure. However, current artificial intelligence approaches typically rely on either global image classification or local object detection, both of which have inherent vulnerabilities. Image classification provides a general scene overview but often misses fine-grained details essential for identifying specific threats. Conversely, object detection is highly sensitive to environmental factors; if a key object is obscured or poorly lit, the entire detection pipeline can fail.

AVS-Net addresses these limitations by introducing a hybrid framework that utilizes a "Relation Network." By reasoning about the spatial and contextual relationships between detected objects and their environment, the system achieves high accuracy and remarkable robustness, even when individual components underperform.

Methodology: The AVS-Net Architecture

The core of the AVS-Net framework is a three-pronged architecture designed to integrate global context with local object details through relational reasoning.

Overview of the AVS-Net architecture, integrating CNN feature maps and object detection through a Relation Network.

As illustrated in Figure 1, the system consists of three main modules:

  1. Contextual Feature Extraction (CNN Feature Maps): A standard Convolutional Neural Network (CNN) processes the input frame into "i-objects." These are feature maps that capture the background, textures, and general environmental context of the scene.
  2. Local Object Detection: The system employs the YOLO (You Only Look Once) model to identify specific "d-objects," such as individuals or potentially dangerous tools, providing their coordinates and classifications.
  3. Relation Reasoning Module: This module forms the "secret sauce" of the architecture. It constructs triplets $(i_i, i_j, d_k)$ by pairing context features with detected objects. These triplets are processed through a Multi-Layer Perceptron (MLP) that learns to interpret the relationships between them—for example, identifying the danger inherent in a person holding a metallic object in a specific setting. The results are aggregated and passed through a final MLP for classification.

Experimental Results and Robustness

The authors evaluated AVS-Net against several baseline models, including Support Vector Machines (SVM), Decision Trees, and Random Forests, using a custom dataset featuring four situational classes: Normal, Danger Tool, Potential Crime, and Dangerous.

Performance comparison showing AVS-Net's robustness as object detection accuracy decreases.

A key finding of the study is the system's "graceful degradation" in performance. As shown in Figure 2, when object detection is 100% accurate, AVS-Net achieves a top-tier accuracy of 96%. However, the true advantage of the Relation Network becomes apparent as detection accuracy is intentionally degraded. When the detector's accuracy drops to 70%, traditional methods like Decision Trees see their performance plummet to 70%. In contrast, AVS-Net maintains a robust 88% accuracy. This demonstrates that the relational reasoning component can effectively infer situational danger even when specific objects are missed by the detector.

Conclusion and Future Work

AVS-Net demonstrates that relational reasoning acts as a vital safety net for automated surveillance. By combining global feature maps with local detections, the system can maintain high performance in suboptimal conditions.

While promising, the current study is limited by a relatively small dataset of 8 scenes and 2 actors. Future research will focus on expanding the dataset through synthetic data generation and real-world testing in diverse CCTV environments to account for variables like weather, lighting, and crowd density. Additionally, optimizing the computational overhead of processing relational triplets will be essential for real-time deployment on edge devices.

Summary updated

BibTeX
@inproceedings{jang2019avs,
  author = {Sein Jang and Young-Ho Park and Aziz Nasridinov},
  title = {{AVS-Net: Automatic Visual Surveillance Using Relation Network}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33019947},
}
Figure