Exploiting Multimodal Interaction Techniques for Video-Surveillance

Marc Castelló, Jordi Gonzàlez 0001, Ariel Amato, Pau Baiget, Carles Fernández, Josep M. Gonfaus, Ramón Alberto Mollineda Cárdenas, Marco Pedersoli, Nicolas Pérez de la Blanca, F. Xavier Roca, Ramón A. Mollineda, Jordi GonzàlezMultimodal Interaction in Image and Video Applications 2013 · DOI · DBLP

Abstract

Conventional video surveillance (VS) systems typically function as passive recording tools, requiring constant human monitoring to interpret scene activity. This paper introduces the VID-Hum prototype, a Cognitive Video Surveillance (CVS) framework designed to transform surveillance into an active "cognitive" assistant. By integrating trajectory-based anomaly detection with Natural Language Processing (NLP) and 3D virtual avatars, VID-Hum enables a multimodal interface where operators can interact with data through conversational commands and receive intuitive, spoken reports.

Introduction

Standard VS systems often suffer from two primary limitations: the difficulty of programmatically defining "normal" behavior and the communication gap between raw quantitative data (e.g., coordinates) and qualitative human understanding. The proposed CVS framework addresses these by translating low-level pixel data into high-level behavioral descriptions, facilitating more effective human-machine collaboration in security contexts.

Methods

The VID-Hum prototype is built upon three pillars of Multimodal Interaction (MI):

  1. Anomaly Detection: The system learns "normal" behavior by tracking trajectories over time using Gaussian Mixture Models (GMMs). It classifies deviations into Soft (speed/orientation changes), Intermediate (path switching), and Hard (unseen paths) anomalies.
  2. Natural Language Understanding (NLU): Users can issue verbal or typed commands, which the NLU module translates into executable logic predicates.
  3. Feedback Generation: Tracking data is converted into natural language sentences via a Natural Language Generation (NLG) module. These descriptions are delivered by XfacePlayer, a 3D virtual avatar that utilizes speech synthesis and facial animation to provide real-time situational reports.
Target tracking module identifying a pedestrian and generating a trajectory for behavioral analysis.

System Architecture and Deployment

The system was deployed in a real-world setting at the Centre de Visió per Computador (CVC) in Barcelona, utilizing a rooftop camera setup (fixed and PTZ) and a cluster of dedicated servers. A central MySQL database serves as the communication backbone, allowing for a modular design where components like the tracker or reasoning engine can be updated independently. This architecture successfully decouples high-level cognitive tasks from specific camera hardware using the Player/Stage system.

The VID-Hum system interface, showing the HERMES Demonstrator (i) and the 3D virtual avatar feedback window (ii).

Key Insights

  • Anomaly as a Spectrum: Categorizing anomalies (Soft, Intermediate, Hard) allows operators to prioritize responses based on the severity of the deviation rather than a binary alert system.
  • Language-Centric Interface: Using natural language as the primary interface serves as a benchmark for the system's "understanding" of the scene, moving beyond simple motion detection to behavioral reasoning.
  • Situation Graph Trees (SGTs): The system uses SGTs to interpret complex pedestrian interactions and generate accurate descriptive text.

Results and Future Work

The VID-Hum prototype successfully demonstrates a closed-loop system capable of interpreting complex queries and generating descriptive text in real-time. While effective in campus environments, the system currently faces challenges in high-density crowds where trajectories become entangled and occlusions are frequent. Future research will focus on the "affective" impact of the virtual avatar, exploring whether an embodied agent improves operator performance and response speed compared to traditional text-based alerts.

Summary updated

BibTeX
@inproceedings{castello2013exploiting,
  author = {Marc Castelló and Jordi Gonzàlez and Ariel Amato and Pau Baiget and Carles Fernández and Josep M. Gonfaus and Ramón Alberto Mollineda Cárdenas and Marco Pedersoli and Nicolas Pérez de la Blanca and F. Xavier Roca and Marc Castelló and Pau Baiget and Ramón A. Mollineda and Nicolás Pérez de la Blanca and Ariel Amato and Carles Fernández and Jordi Gonzàlez and Josep M. Gonfaus},
  title = {{Exploiting Multimodal Interaction Techniques for Video-Surveillance}},
  booktitle = {Multimodal Interaction in Image and Video Applications},
  year = {2013},
  doi = {10.1007/978-3-642-35932-3\_8},
}
Figure