Multi-View Surveillance Video Summarization via Joint Embedding and Sparse Optimization

Rameswar Panda, Amit K. Roy-Chowdhury

TL;DR

Summarizing footage from multiple overlapping security cameras is challenging due to redundant angles, unaligned feeds, and massive data volumes. This paper introduces an unsupervised framework that jointly learns a latent embedding of the multi-view videos and selects a sparse set of representative shots. By optimizing both steps together, the method generates highly informative, concise summaries without requiring any prior camera calibration or synchronization.

What Problem Does This Paper Address?

Video summarization is a well-studied problem, but most existing methods are designed for single-view videos. When you apply these methods to a network of surveillance cameras (multi-view videos), they fail to account for the complex spatial and temporal correlations across different cameras.

In a multi-camera setup, fields of view often overlap, meaning the same event might be captured from multiple angles with varying illumination, poses, and depth of field. Furthermore, these camera networks are often uncalibrated and unsynchronized. The challenge is to sift through this massive, unaligned data to extract a single, concise video skim (a sequence of short shots) that captures the most important events across the entire network without overwhelming the viewer with redundant footage.

Methods

The authors propose an unsupervised approach that frames multi-view summarization as a joint optimization problem combining manifold embedding and sparse representative selection.

Here is how the pipeline works:

  1. Video Representation: Videos are first segmented into non-uniform shots based on color space changes. To capture the temporal dynamics of surveillance footage, the authors extract spatio-temporal C3D features (using a 3D Convolutional Neural Network) for each shot, rather than relying on static 2D frames.
  2. Multi-view Video Embedding: To handle uncalibrated cameras, the method projects the high-dimensional shot features into a lower-dimensional latent space. It uses an $\ell_1$-norm sparse coding technique to build a similarity graph that preserves both intra-view (within the same camera) and inter-view (across different cameras) correlations.
  3. Sparse Representative Selection: To pick the actual summary shots, the authors apply an $\ell_{2,1}$-norm regularization. Unlike the $\ell_1$-norm which induces element-wise sparsity, the $\ell_{2,1}$-norm induces row-level sparsity in the coefficient matrix. The non-zero rows directly correspond to the most informative shots to include in the summary.
  4. Joint Optimization: Instead of doing embedding and selection sequentially, the authors combine them into a single non-smooth, non-convex objective function. They solve this efficiently using an alternating algorithm based on half-quadratic minimization.

Datasets

The authors evaluated their approach on 6 standard multi-view video datasets (comprising 36 videos total) captured in both indoor and outdoor environments:

  • Office, Campus, Lobby, Road, Badminton: Standard benchmarks from prior works, featuring 3 to 4 overlapping camera views.
  • BL-7F: A large-scale, highly complex dataset captured with 19 surveillance cameras on the 7th floor of a university building.

Key Insights and Ideas

  • Joint optimization is a game-changer: By adaptively changing the embedding graph based on the sparse selection process, the latent space doesn't just map the video structure—it actively shapes itself to highlight the best representative shots. This joint approach yielded a ~3% F-measure boost over doing the steps sequentially.
  • "Analyze once, generate many": Because the $\ell_{2,1}$ optimization assigns a continuous importance weight to every shot, the system is highly scalable. Users can request a 3-shot summary, a 5-shot summary, or a 10-shot summary on the fly simply by taking the top $N$ weighted shots. Competing methods usually require completely recalculating the summary if the desired length changes.
  • Motion matters: Using spatio-temporal C3D features outperformed traditional 2D frame-level features (like VGG-net) by about 2.5% in F-measure, proving that the temporal aspect of activities is crucial for understanding surveillance footage.

Key Results

  • Quantitative Superiority: The proposed method significantly outperformed state-of-the-art multi-view summarization techniques (like RandomWalk, RoughSets, and BipartiteOPF). On the massive 19-camera BL-7F dataset, the approach achieved an F-measure of 71.29%, crushing the previous baseline (GMM) which scored 60.00%.
  • High Precision and Recall: On the Office and Lobby datasets, the method achieved F-measures of 89.36% and 92.52%, respectively. It successfully captured events (like a person looking for a book in a cubicle) that competing methods entirely missed.
  • Qualitative User Study: In a blind test with 5 human experts rating summaries on a scale of 1 to 10 for visual pleasantness and information content, the proposed method consistently ranked first (e.g., scoring 8.2/10 on the Lobby dataset compared to the next best method's 7.4/10).

Limitations and Future Work

  • Rigid Preprocessing: The temporal segmentation (cutting the video into shots) is currently a fixed preprocessing step. If the segmentation algorithm makes a mistake, the summarizer inherits it. The authors suggest that future work could use Markov Decision Processes (MDPs) to dynamically adjust shot boundaries during the summarization process.
  • Redundant Angles: Because the algorithm prioritizes minimizing reconstruction error, it sometimes selects multiple shots of the same event from different camera angles. While helpful for security analysis, this inflates the summary length.
  • Lack of High-Level Semantics: The current model relies purely on feature-level correlations. Future iterations could incorporate semantic feature analysis or attention modeling to prioritize specific types of events based on user preferences.

Summary updated

Abstract

Most traditional video summarization methods are designed to generate effective summaries for single-view videos, and thus, they cannot fully exploit the complicated intra- and inter-view correlations in summarizing multi-view videos in a camera network. In this paper, with the aim of summarizing multi-view videos, we introduce a novel unsupervised framework via joint embedding and sparse representative selection. The objective function is twofold. The first is to capture the multiview correlations via an embedding, which helps in extracting a diverse set of representatives. The second is to use a ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,1</sub> -norm to model the sparsity while selecting representative shots for the summary. We propose to jointly optimize both of the objectives, such that embedding cannot only characterize the correlations, but also indicate the requirements of sparse representative selection. We present an efficient alternating algorithm based on half-quadratic minimization to solve the proposed non-smooth and non-convex objective with convergence analysis. A key advantage of the proposed approach with respect to the state-of-the-art is that it can summarize multi-view videos without assuming any prior correspondences/alignment between them, e.g., uncalibrated camera networks. Rigorous experiments on several multi-view datasets demonstrate that our approach clearly outperforms the state-of-the-art methods.
BibTeX
@article{panda2017multi,
  author = {Rameswar Panda and Amit K. Roy-Chowdhury},
  title = {{Multi-View Surveillance Video Summarization via Joint Embedding and Sparse Optimization}},
  journal = {IEEE Trans. Multimedia},
  year = {2017},
  url = {https://openreview.net/forum?id=Jqa_lZQ8Smi},
}

Loading related content...

Figure