Multi-View Surveillance Video Summarization via Joint Embedding and Sparse Optimization

Rameswar Panda, Amit K. Roy-Chowdhury

Abstract

Most traditional video summarization methods are designed to generate effective summaries for single-view videos, and thus they cannot fully exploit the complicated intra and inter-view correlations in summarizing multi-view videos in a camera network. In this paper, with the aim of summarizing multi-view videos, we introduce a novel unsupervised framework via joint embedding and sparse representative selection. The objective function is two-fold. The first is to capture the multi-view correlations via an embedding, which helps in extracting a diverse set of representatives. The second is to use a $\ell_{2,1}$-norm to model the sparsity while selecting representative shots for the summary. We propose to jointly optimize both of the objectives, such that embedding can not only characterize the correlations, but also indicate the requirements of sparse representative selection. We present an efficient alternating algorithm based on half-quadratic minimization to solve the proposed non-smooth and non-convex objective with convergence analysis. A key advantage of the proposed approach with respect to the state-of-the-art is that it can summarize multi-view videos without assuming any prior correspondences/alignment between them, e.g., uncalibrated camera networks. Rigorous experiments on several multi-view datasets demonstrate that our approach clearly outperforms the state-of-the-art methods.

Introduction

Summarizing footage from multiple overlapping security cameras is challenging due to redundant angles, unaligned feeds, and massive data volumes. While video summarization is well-studied for single-view videos, traditional methods often fail to account for the complex spatial and temporal correlations across different cameras in a network.

A multi-camera surveillance network with overlapping fields of view (C1-C6) and sample frames.

In a multi-camera setup, fields of view often overlap, meaning the same event might be captured from multiple angles with varying illumination, poses, and depth of field. Furthermore, these camera networks are often uncalibrated and unsynchronized. The challenge is to sift through this massive, unaligned data to extract a single, concise video skim that captures the most important events across the entire network without overwhelming the viewer with redundant footage.

Methodology

The proposed unsupervised framework frames multi-view summarization as a joint optimization problem combining manifold embedding and sparse representative selection.

  1. Video Representation: Videos are segmented into non-uniform shots based on color space changes. Spatio-temporal C3D features (using a 3D CNN) are extracted for each shot to capture temporal dynamics.
  2. Multi-view Video Embedding: To handle uncalibrated cameras, high-dimensional shot features are projected into a lower-dimensional latent space. An $\ell_1$-norm sparse coding technique builds a similarity graph preserving both intra-view and inter-view correlations.
  3. Sparse Representative Selection: An $\ell_{2,1}$-norm regularization is applied to pick summary shots. This induces row-level sparsity in the coefficient matrix, where non-zero rows correspond to the most informative shots.
  4. Joint Optimization: Embedding and selection are combined into a single non-smooth, non-convex objective function, solved efficiently using an alternating algorithm based on half-quadratic minimization.

Experimental Setup

The approach was evaluated on six standard multi-view video datasets (36 videos total) captured in both indoor and outdoor environments:

  • Office, Campus, Lobby, Road, Badminton: Benchmarks featuring 3 to 4 overlapping camera views.
  • BL-7F: A large-scale dataset with 19 surveillance cameras on a university building floor.

Results and Discussion

The joint optimization approach adaptively changes the embedding graph based on the selection process, yielding a ~3% F-measure boost over sequential processing. Additionally, using spatio-temporal C3D features outperformed traditional 2D features by 2.5%, highlighting the importance of motion in understanding surveillance footage.

Qualitative comparison showing the proposed method's ability to capture specific events (highlighted in red) that other methods might miss.

Quantitatively, the method significantly outperformed state-of-the-art techniques like RandomWalk and BipartiteOPF. On the BL-7F dataset, it achieved an F-measure of 71.29%, surpassing the 60.00% baseline. Qualitatively, user studies ranked the method first in visual pleasantness and information content, successfully capturing events like a person looking for a book in a cubicle that competing methods missed.

Conclusion

While effective, the method currently relies on fixed temporal segmentation. Future work may explore dynamic shot boundaries using Markov Decision Processes. Additionally, incorporating high-level semantics could help prioritize specific event types based on user preferences, further reducing redundancy in multi-angle captures.

Summary updated

Abstract

Most traditional video summarization methods are designed to generate effective summaries for single-view videos, and thus they cannot fully exploit the complicated intra and inter-view correlations in summarizing multi-view videos in a camera network. In this paper, with the aim of summarizing multi-view videos, we introduce a novel unsupervised framework via joint embedding and sparse representative selection. The objective function is two-fold. The first is to capture the multi-view correlations via an embedding, which helps in extracting a diverse set of representatives. The second is to use a `2;1- norm to model the sparsity while selecting representative shots for the summary. We propose to jointly optimize both of the objectives, such that embedding can not only characterize the correlations, but also indicate the requirements of sparse representative selection. We present an efficient alternating algorithm based on half-quadratic minimization to solve the proposed non-smooth and non-convex objective with convergence analysis. A key advantage of the proposed approach with respect to the state-of-the-art is that it can summarize multi-view videos without assuming any prior correspondences/alignment between them, e.g., uncalibrated camera networks. Rigorous experiments on several multi-view datasets demonstrate that our approach clearly outperforms the state-of-the-art methods.
BibTeX
@article{panda2017multi,
  author = {Rameswar Panda and Amit K. Roy-Chowdhury},
  title = {{Multi-View Surveillance Video Summarization via Joint Embedding and Sparse Optimization}},
  journal = {IEEE Trans. Multimedia},
  year = {2017},
  url = {https://openreview.net/forum?id=Jqa_lZQ8Smi},
}
Figure