Automatic Association of Chats and Video Tracks for Activity Learning and Recognition in Aerial Video Surveillance

Riad I. Hammoud, Cem S. Sahin, Erik Blasch, Bradley J. Rhodes, Tao Wang

TL;DR

This paper introduces a novel framework for automatically associating human-generated chat messages with video tracks in aerial surveillance footage. By fusing these two data streams, the system can learn and recognize complex activities, significantly improving the efficiency and accuracy of video analysis for intelligence and surveillance applications.

What Problem Does This Paper Address?

Analyzing vast amounts of aerial surveillance video (Full-Motion Video or FMV) is a time-consuming and labor-intensive task. Analysts often manually review footage, identifying and annotating "Activities of Interest" (AOIs) and "Targets of Interest" (TOIs). While analysts may verbally describe these events in chat messages, this information is typically not synchronized with the video, making it difficult to automatically link specific chat descriptions to corresponding video events. This disconnect hinders efficient searching, indexing, and learning of activity patterns from large video archives. The paper aims to bridge this gap by automatically associating unstructured chat data with structured video track data to enable automated activity recognition and intelligent video indexing.

Methods

The core of the proposed system lies in two main components: VIVA (Video-Indexed by Voice Annotations) and MINER (Multi-Media Indexing and Explorer).

  • VIVA focuses on processing the aerial video. It includes:
  • Video Stabilization: Compensating for camera motion to ensure stable video frames.
  • Target Detection and Tracking (COURSE): Identifying and tracking moving objects in the stabilized video. This algorithm is designed to be robust to challenging conditions like low resolution and clutter.
  • Target Classification: Categorizing detected targets into "person," "vehicle," or "others" using Support Vector Machines (SVMs) with HOG (Histogram of Oriented Gradients) and HOF (Histogram of Optical Flow) features.
  • Track Segmentation: Dividing long, continuous tracks into shorter "semantic segments" based on significant changes in attributes like direction and speed. This allows for more granular analysis of activities.
  • MINER handles the chat messages and the association process:
  • Chat Message Parsing: Extracting key attributes (activity type, direction, location, time, etc.) from chat messages.
  • Graph Representation: Both video tracks (specifically, the semantic segments) and chat messages are represented as graphs, where nodes represent attributes and edges represent relationships.
  • Multi-Source Graph Association: A probabilistic graph-matching framework is used to find the best correspondence between chat message graphs and video track segment graphs. This framework accounts for missing attributes and potential errors in both data sources by using weighted attribute probabilities.
  • Activity Pattern Learning (MAPLE): The associated chat-video pairs serve as training data to learn activity patterns using an unsupervised clustering algorithm (HELM). This learned model can then be used to index and recognize activities in unlabeled video data.

Datasets

The paper utilizes a custom dataset consisting of:

  • EO/IR (Electro-Optical/Infrared) airborne videos (60 minutes long).
  • 100 chat messages.

The authors also mention using sample videos from the VIRAT aerial dataset (ApHill) for demonstrating track generation and activity pattern matching.

Key Insights and Ideas

  • Chat as Ground Truth: The paper highlights that analyst chat messages, despite being unstructured, can serve as a valuable source of "pseudo ground truth" for activity recognition in surveillance video.
  • Semantic Segmentation of Tracks: Breaking down long video tracks into shorter, semantically meaningful segments is crucial for accurately associating them with specific chat descriptions of activities.
  • Probabilistic Graph Matching: Using a probabilistic approach for graph association allows for robust matching even when attributes are missing or noisy, which is common in real-world surveillance data.
  • Paradigm Shift in Training Data: The framework demonstrates a shift from requiring manually annotated video for training to leveraging existing analyst chat logs, significantly reducing the annotation burden.
  • Multi-Modal Fusion for Context: The fusion of visual data (tracks) and textual data (chats) provides richer context for understanding activities than either modality alone.

Key Results

  • Association Accuracy: The multi-graph association framework achieved 83.77% correct association, with 6.39% misses and 9.84% false alarms on their dataset.
  • Target Classification: Track-wise classification accuracy reached 86.58% (HOF) and 92.61% (HOG) for distinguishing between "person," "vehicle," and "others."
  • Activity Classification (Unlabeled Data): The system achieved 79.6% correct activity label assignment for unlabeled tracks, demonstrating the effectiveness of the learned activity patterns.
  • Improved Tracking: The VIVA system showed improvement in tracking compared to video data alone, achieving 84% detection with modest misdetection/false alarm rates.

Limitations and Future Work

  • Noise in Training Data: The automatically generated training data from the multi-source association framework is not noise-free, with approximately 10% false classification. The authors suggest that human-in-the-loop correction of training data would improve pattern learning.
  • Attribute Limitations: While direction and location are important, the authors note that additional attributes like shape, color, size, and spatial relationships to landmarks would further reduce association ambiguities.
  • Chat Description Granularity: Chat descriptions are often cast to gross zones (e.g., "middle screen region," "northeast direction"), leading to ambiguities. Extracting more detailed information, such as building references, could enhance association.
  • Future Work: The authors plan to explore metrics and associations to increase robustness and reduce false alarms. They also envision end-users being able to check results in their MINER interface to remove false alarms and generate mission reports. Further work could also involve exploring more sophisticated NLP techniques for chat parsing and incorporating more diverse data modalities.

Summary updated

Abstract

We describe two advanced video analysis techniques, including video-indexed by voice annotations (VIVA) and multi-media indexing and explorer (MINER). VIVA utilizes analyst call-outs (ACOs) in the form of chat messages (voice-to-text) to associate labels with video target tracks, to designate spatial-temporal activity boundaries and to augment video tracking in challenging scenarios. Challenging scenarios include low-resolution sensors, moving targets and target trajectories obscured by natural and man-made clutter. MINER includes: (1) a fusion of graphical track and text data using probabilistic methods; (2) an activity pattern learning framework to support querying an index of activities of interest (AOIs) and targets of interest (TOIs) by movement type and geolocation; and (3) a user interface to support streaming multi-intelligence data processing. We also present an activity pattern learning framework that uses the multi-source associated data as training to index a large archive of full-motion videos (FMV). VIVA and MINER examples are demonstrated for wide aerial/overhead imagery over common data sets affording an improvement in tracking from video data alone, leading to 84% detection with modest misdetection/false alarm results due to the complexity of the scenario. The novel use of ACOs and chat Sensors 2014, 14 19844 messages in video tracking paves the way for user interaction, correction and preparation of situation awareness reports.
BibTeX
@inproceedings{hammoud2014automatic,
  author = {Riad I. Hammoud and Cem S. Sahin and Erik Blasch and Bradley J. Rhodes and Tao Wang},
  title = {{Automatic Association of Chats and Video Tracks for Activity Learning and Recognition in Aerial Video Surveillance}},
  booktitle = {Sensors},
  year = {2014},
  url = {https://openreview.net/forum?id=in_WyHVndUz},
}

Loading related content...

Figure