Influence of Video Dynamics on EEG-based Single-Trial Video Target Surveillance System

Heon-Gyu Kwak, Sung-Jin Kim, Hyeon-Taek Han, Ji-Hoon Jeong, Seong-Whan LeearXiv 2023 · arXiv

TL;DR

This paper explores using electroencephalography (EEG) as a supportive module for military video surveillance, leveraging human brain signals to detect targets when training data for computer vision models is scarce. While the proposed hierarchical deep learning framework performs well on static video footage, the authors discover that dynamic video elements—like camera rotations and weather changes—inadvertently bias the model toward passive visual processing rather than active target detection.

What Problem Does This Paper Address?

Conventional computer vision (CV) models excel at target detection, but they face a major bottleneck in specialized fields like military surveillance: a severe lack of real-world training samples for hostile targets. To bridge this gap, researchers are turning to Brain-Computer Interfaces (BCIs). By monitoring a human operator's brainwaves (specifically via EEG), a system can detect the neural signatures of target recognition, effectively using human perception as a zero-shot detector.

However, real-world surveillance video isn't static. Cameras pan, tilt, and zoom, and weather conditions shift. This paper investigates a critical, previously overlooked question: How do video dynamics (like camera rotation and environmental changes) affect the performance of an EEG-based target detection model?

Methods

The authors developed an asynchronous, real-time BCI framework for video target detection using a modified version of DeepConvNet, a popular convolutional neural network architecture for EEG data.

To optimize performance, they introduced two key modifications:

  1. Hierarchical Architecture: Instead of classifying everything at once, the model operates in two stages. Stage 1 performs binary classification to separate "non-target" from "target" states. If a target is detected, Stage 2 re-classifies it into a "true-target" (e.g., a hostile soldier) or an "error-target" (e.g., a non-hostile animal).
  2. Data Augmentation for Extreme Imbalance: In surveillance, targets appear rarely, creating a massive class imbalance (e.g., 87.5% non-target data). The authors used a sliding window approach (250 time points, 25-point stride) to augment the minority target classes tenfold.

The model was pre-trained on a multi-subject dataset and fine-tuned for individual subjects during online testing.

Subject undergoing EEG data acquisition during video surveillance simulation.

Datasets

The researchers generated simulated video surveillance footage set in a mountainous forest area. They created three distinct video stimuli to isolate the impact of video dynamics:

  • Video1: A 300-second static video with no camera movement or weather changes.
  • Video2-N: A 480-second dynamic video where the camera periodically rotates for 3 seconds every 5 seconds, and the weather gradually transitions to foggy or rainy.
  • Video2-AI: Identical to Video2-N, but includes a red bounding box around targets to simulate an AI assistant guiding the user's attention.

In all videos, three targets suddenly appear for one second: 'deer' and 'wild boar' (error-targets), and 'soldier' (true-target). Data was collected from 13 healthy subjects (4 offline for training, 9 online for testing).

Fig. 1. An overview of the EEG data acquisition process. A subject first watches ‘Video1’ twice, and then watches ‘Video2-N’ and ‘Video2-AI’ twice in turn. Each video clip is eight minutes long. The subject takes rests for two minutes after a session ends. Each subject proceeds four sessions of EEG acquisition, taking 52 minutes in total.

Key Insights and Ideas

  • The "Passive Visual" Distraction: The most striking takeaway is that video dynamics break the target detection model. When a camera rotates or weather changes, it triggers strong, passive visual processing in the brain. The deep learning model latches onto these prominent neural signatures instead of the subtler "aha!" signals associated with active target detection.
  • Saliency Map Revelations: By using saliency maps to visualize which EEG channels mattered most to the network, the authors found that models trained on dynamic videos relied heavily on the central, temporal, and occipital regions. These areas are responsible for passive visual perception (processing motion and scene changes) rather than the frontal and temporal-parietal networks typically engaged during discriminative, task-driven target detection.
Channel-wise saliency maps showing model reliance on passive visual areas (deep blue) rather than task-related frontal regions.

Key Results

The performance gap between static and dynamic surveillance conditions was stark, measured using the macro $F_\beta$ metric (with $\beta=2$ to prioritize recall over precision):

  • Static Environment (Video1): Achieved a promising mean macro $F_\beta$ of 0.6522, with some subjects scoring as high as 0.7568.
  • Dynamic Environment (Video2-N): Performance plummeted to a mean macro $F_\beta$ of 0.1282.
  • Dynamic Environment with AI Assistance (Video2-AI): Even with a red bounding box to guide attention, the model failed, yielding a mean macro $F_\beta$ of just 0.1007.

Event-Related Potential (ERP) analysis confirmed that for the dynamic videos, the temporal patterns of target recognition were obscured by the brain's response to background camera movement.

Limitations and Future Work

The primary limitation is the model's current inability to filter out environmental noise and camera motion from the EEG signal. If a BCI target detector is to be deployed in real-world, moving surveillance systems, it cannot be fragile to basic camera panning.

Future Directions: The authors emphasize that future BCI paradigms must be designed to decouple passive visual perception features from active, task-related features. This could involve developing advanced neural filtering techniques or training models specifically to ignore motion-induced artifacts in EEG data.

Summary updated

Abstract

Target detection models are one of the widely used deep learning-based applications for reducing human efforts on video surveillance and patrol. However, the application of conventional computer vision-based target detection models in military usage can result in limited performance, due to the lack of sample data of hostile targets. In this paper, we present the possibility of the electroencephalography-based video target detection model, which could be applied as a supportive module of the military video surveillance system. The proposed framework and detection model showed prospective performance achieving a mean macro F-beta of 0.6522 with asynchronous real-time data from five subjects, in a certain video stimulus, but not on some video stimuli. By analyzing the results of experiments using each video stimulus, we present the factors that would affect the performance of electroencephalography-based video target detection models.
BibTeX
@misc{kwak2023influence,
  author = {Heon-Gyu Kwak and Sung-Jin Kim and Hyeon-Taek Han and Ji-Hoon Jeong and Seong-Whan Lee},
  title = {{Influence of Video Dynamics on EEG-based Single-Trial Video Target Surveillance System}},
  year = {2023},
  eprint = {2311.08631},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.08631},
}
Figure