Probabilistic Semantic Retrieval for Surveillance Videos with Activity Graphs

Yuting Chen, Joseph Wang, Yannan Bai, Gregory D. Castañón, Venkatesh Saligrama

Abstract

We present a novel framework for finding complex activities matching user-described queries in cluttered surveillance videos. The wide diversity of queries coupled with the unavailability of annotated activity data limits our ability to train activity models. To bridge the semantic gap, we propose letting users describe an activity as a semantic graph with object attributes and inter-object relationships associated with nodes and edges, respectively. We learn node/edge-level visual predictors during training and, at test-time, propose retrieving activity by identifying likely locations that match the semantic graph. We formulate a novel conditional random field-based probabilistic activity localization objective that accounts for misdetections, misclassifications and track losses, and outputs a likelihood score for a candidate grounded location of the query in the video. We seek groundings that maximize overall precision and recall. To handle the combinatorial search over all high-probability groundings, we propose a highest precision subgraph matching algorithm. Our method outperforms existing retrieval methods on benchmarked datasets.

Introduction

Traditional video retrieval usually relies on "query-by-example," where a user provides a video clip to find similar instances. In surveillance contexts, this is often impractical because the specific event of interest may be unique or unrecorded. Furthermore, training deep learning models for every possible activity requires massive amounts of annotated data, which is often unavailable for niche surveillance scenarios.

This paper addresses the "semantic gap"—the disconnect between high-level human descriptions and low-level pixel data. Surveillance video is inherently "noisy": objects are frequently occluded, tracks are lost, and many unrelated activities occur simultaneously. A robust system must understand not just what objects are present, but the spatial and temporal relationships between them, while accounting for the errors of automated sensors.

Methodology: Activity Graphs and Probabilistic Grounding

The proposed "zero-shot" framework does not require training on specific activities. Instead, it decomposes the problem into three primary components:

  1. Activity Graphs: Users describe an activity as a graph where nodes represent objects (e.g., "red car", "person") and edges represent relationships (e.g., "near," "later," or "same ID").
  2. Probabilistic Grounding (CRF): Finding the activity in a video is treated as a "grounding" problem—mapping graph nodes to actual detected video tracklets. A Conditional Random Field (CRF) calculates a likelihood score, allowing for "soft" matches when detectors are uncertain or trackers fail.
  3. Efficient Search (HPST & HPSM): To manage the combinatorial complexity of searching for graph patterns, the authors introduce the Highest Precision Spanning Tree (HPST). By identifying the most discriminative parts of a query first, the system prunes the search space before performing full Highest Precision Subgraph Matching (HPSM).
Overview of the ADSA framework, showing the mapping from a text query to an activity graph and its grounding in video data.

Experimental Evaluation

The framework was evaluated on three distinct types of video data: VIRAT Ground 2.0 (parking lot interactions), AFRL WAMI (wide-area aerial imagery), and AvA (varied movie clips). These datasets test the system's ability to handle everything from standard surveillance to low-resolution, small-object detection in massive areas.

Key Results

The proposed method significantly outperformed baselines. On the VIRAT dataset, it achieved an average Area-Under-Curve (AUC) of 83.72%, compared to 18.58% for a Bag-of-Words (BoW) approach and 74.34% for a previous non-probabilistic graph matching method.

Performance comparison (AUC) of the proposed method against BoW and MSGM baselines across various complex activity queries.

The system proved robust to noise, maintaining its lead even when using automated trackers rather than perfect human annotations. It excelled at complex, multi-stage queries like "car following + stop + dismount," where temporal order and identity persistence are critical.

Discussion and Key Insights

A central finding is that relationships are the "secret sauce" for retrieval. A simple Bag-of-Words approach fails in crowded scenes because it lacks context. By learning relationship concepts (like "near") from data rather than hard-coding them, the system adapts to different environments (e.g., highway vs. hallway). Furthermore, the use of pruning via spanning trees turns a computationally hard problem into one solvable with dynamic programming.

Conclusion

While the system is robust, it remains dependent on the quality of underlying object detectors. Future work aims to bridge the remaining gap between natural language and graph construction, potentially allowing users to search using raw text sentences that the system automatically converts into activity graphs.

Summary updated

Abstract

We present a novel framework for finding complex activities matching user-described queries in cluttered surveillance videos. The wide diversity of queries coupled with unavailability of annotated activity data limits our ability to train activity models. To bridge the semantic gap we propose to let users describe an activity as a semantic graph with object attributes and inter-object relationships associated with nodes and edges, respectively. We learn node/edge-level visual predictors during training and, at test-time, propose to retrieve activity by identifying likely locations that match the semantic graph. We formulate a novel CRF based probabilistic activity localization objective that accounts for mis-detections, mis-classifications and track-losses, and outputs a likelihood score for a candidate grounded location of the query in the video. We seek groundings that maximize overall precision and recall. To handle the combinatorial search over all high-probability groundings, we propose a highest precision subgraph matching algorithm. Our method outperforms existing retrieval methods on benchmarked datasets.
BibTeX
@article{chen2017probabilistic,
  author = {Yuting Chen and Joseph Wang and Yannan Bai and Gregory D. Castañón and Venkatesh Saligrama},
  title = {{Probabilistic Semantic Retrieval for Surveillance Videos with Activity Graphs}},
  journal = {IEEE Transactions on Multimedia},
  year = {2017},
  doi = {10.1109/tmm.2018.2865860},
}
Figure