Discovery of Shared Semantic Spaces for Multi-Scene Video Query and Summarization

Xun Xu, Timothy M. Hospedales, Shaogang Gong

Abstract

Automated surveillance systems typically analyze video feeds in isolation, failing to exploit semantic similarities between related scenes. This research introduces a framework for multi-scene global understanding that clusters surveillance scenes based on their behavioral commonalities. By discovering a Shared Topic Basis (STB) within these clusters, the system creates a unified activity vocabulary. This structured representation improves key surveillance tasks, including cross-scene query-by-example, behavior classification with minimal labeling, and multi-scene video summarization. Evaluations on a new 27-scene traffic dataset demonstrate that this multi-scene approach significantly outperforms traditional single-scene models.

Introduction

The proliferation of CCTV networks has generated vast amounts of video data, necessitating automated exploitation for scene understanding and summarization. Most existing research focuses on single scenes or adjacent cameras. However, many scenes—such as different traffic intersections—share similar semantic activities despite having distinct pixel-level appearances. Exploiting these similarities is challenging because unrelated scenes can cause "negative transfer" if modeled together. This paper addresses these challenges by selectively sharing information across semantically related scene clusters.

Methods

The proposed framework discovers related scenes and shares information through a multi-layer process:

  1. Local Activity Learning: Optical flow is extracted and quantized into "visual words." Latent Dirichlet Allocation (LDA) then identifies local "topics" or activities within each scene.
  2. Scene Alignment & Clustering: Scenes are aligned geometrically and compared using Kullback-Leibler (KL) divergence of their local topics. Spectral clustering then groups semantically related scenes.
  3. Shared Topic Basis (STB): Within each cluster, local topics are projected into a reference coordinate system to form a unified vocabulary of behaviors.
  4. Downstream Applications: Video clips are represented as profiles over the STB, enabling cross-scene queries, zero-shot classification, and global summarization.
The multi-layer framework for multi-scene global understanding, from local activity extraction to downstream applications.

Datasets

To evaluate the framework, the authors introduced a new dataset comprising 27 surveillance scenes. This includes 25 real-world traffic feeds from Budapest, Hungary, and two established datasets (Junction and Roundabout). A subset of 672 clips across 6 scenes was exhaustively annotated with behavior tags to test classification and summarization performance.

Overview of the 27 traffic surveillance scenes used to validate the multi-scene framework.

Key Insights and Results

  • Activity-Based Similarity: Grouping scenes by dynamic motion topics is more robust than pixel-based matching, as it ignores background distractors like lighting and architecture.
  • Selective Sharing: The Scene Cluster Model (SCM) avoids the pitfalls of a "Flat Model" (which treats all scenes as one) by only sharing information within relevant clusters. SCM achieved 42.56% accuracy in cross-scene classification, outperforming the flat baseline.
  • Efficient Summarization: Summarizing across a cluster allows for higher compression. The system can capture the majority of unique behaviors across multiple cameras in a single, short video skim, reducing nearly 90 minutes of footage to a few representative minutes.

Conclusion

By discovering shared semantic spaces, this framework enables surveillance networks to function as a cohesive system rather than a collection of isolated sensors. Future work will focus on handling more complex pedestrian behaviors and developing online learning methods for real-time deployment.

Summary updated

Abstract

The growing rate of public space CCTV installations has generated a need for automated methods for exploiting video surveillance data including scene understanding, query, behaviour annotation and summarization. For this reason, extensive research has been performed on surveillance scene understanding and analysis. However, most studies have considered single scenes, or groups of adjacent scenes. The semantic similarity between different but related scenes (e.g., many different traffic scenes of similar layout) is not generally exploited to improve any automated surveillance tasks and reduce manual effort. Exploiting commonality, and sharing any supervised annotations, between different scenes is however challenging due to: Some scenes are totally un-related -- and thus any information sharing between them would be detrimental; while others may only share a subset of common activities -- and thus information sharing is only useful if it is selective. Moreover, semantically similar activities which should be modelled together and shared across scenes may have quite different pixel-level appearance in each scene. To address these issues we develop a new framework for distributed multiple-scene global understanding that clusters surveillance scenes by their ability to explain each other's behaviours; and further discovers which subset of activities are shared versus scene-specific within each cluster. We show how to use this structured representation of multiple scenes to improve common surveillance tasks including scene activity understanding, cross-scene query-by-example, behaviour classification with reduced supervised labelling requirements, and video summarization. In each case we demonstrate how our multi-scene model improves on a collection of standard single scene models and a flat model of all scenes.
BibTeX
@article{xu2017discovery,
  author = {Xun Xu and Timothy M. Hospedales and Shaogang Gong},
  title = {{Discovery of Shared Semantic Spaces for Multi-Scene Video Query and Summarization}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2017},
  doi = {10.1109/tcsvt.2016.2532719},
}
Figure