WATTS: a Web Annotation Tool for Surveillance Scenarios

Federico Bartoli, Lorenzo Seidenari, Giuseppe Lisanti, Svebor Karaman, Alberto Del BimboACM Multimedia 2015 · DOI

TL;DR

This paper introduces WATSS, a web-based annotation tool designed for creating detailed ground truth for surveillance datasets. WATSS enables collaborative annotation of person location, identity, gaze, and occlusion, addressing the growing need for rich datasets to advance behavior understanding in computer vision.

What Problem Does This Paper Address?

The field of computer vision, particularly in areas like object detection and behavior understanding, relies heavily on large, well-annotated datasets. While datasets like ImageNet have driven progress in image recognition, there's a growing need for more complex datasets that capture nuanced human behaviors in real-world scenarios, such as those found in surveillance footage. Specifically, understanding group dynamics, individual interactions, and how people behave when partially obscured requires annotations beyond simple bounding boxes. Existing annotation tools often lack the flexibility to capture this diverse information or are not designed for collaborative, web-based annotation, making the creation of such datasets time-consuming and expensive.

Methods

WATSS is a web application built using HTML5, JavaScript, and CSS, with its data persistence managed by a relational database. Its core innovation lies in its comprehensive annotation capabilities and user-friendly interface designed for collaborative use. Key features include:

  • Multi-camera dataset management: The tool can handle data from multiple camera viewpoints simultaneously.
  • Detailed annotation types: Annotators can mark:
  • Person location: Using bounding boxes.
  • Visible area: A secondary bounding box to denote the non-occluded portion of a person.
  • Person identity: Assigning a unique, persistent ID across frames and cameras.
  • Gaze and body orientation: Quantified with 5-degree precision.
  • Group membership: Assigning individuals to groups with persistent IDs.
  • Collaborative annotation: The web-based nature allows multiple users to annotate concurrently.
  • Usability enhancements: Features like "Add person" interface showing existing identities, suggestions for bounding boxes and gaze in subsequent frames, and keyboard shortcuts aim to streamline the annotation process.

Datasets

The paper introduces the MuseumVisitors dataset as a prime example of the type of data WATSS is designed to annotate. This dataset, recorded at the National Museum of Bargello in Florence, Italy, focuses on pedestrian and group detection, gaze estimation, and behavior understanding. The authors also mention using this dataset for evaluation, annotating 96,972 detections, gazes, and 101 person identities over 9,477 frames from four cameras.

Key Insights and Ideas

  • The necessity of rich annotations for behavior understanding: The paper strongly argues that to move beyond basic detection and truly understand complex human behaviors (like group interactions or responses to occlusion), datasets need to capture a wider range of information than traditional object detection datasets.
  • Web-based collaboration is key for large-scale annotation: The authors highlight that for complex annotation tasks, a web-based platform is crucial for enabling multiple annotators to work efficiently and concurrently, significantly reducing the time and cost associated with dataset creation.
  • Bridging the gap between detection and behavior: WATSS aims to fill a void by providing a tool that can generate annotations necessary for both detection tasks and more advanced behavior analysis, which often require information like gaze and occlusion status.
  • Iterative annotation improves efficiency: The feature that carries over bounding boxes and annotations to subsequent frames is a practical insight for speeding up the tedious process of video annotation.

Key Results

The paper reports a usability study using the System Usability Scale (SUS), where WATSS achieved an average score of 70. Key findings from the usability study include:

  • Users generally found the system easy to learn and use.
  • Annotators felt confident and productive using the system.
  • A neutral response to item 6 ("I found the various functions in this system were well integrated") suggests room for improvement, possibly due to the diversity of annotations requested.

The authors also mention that WATSS was used to annotate their MuseumVisitors dataset, a task that took approximately 3 person-months for 5 annotators.

Limitations and Future Work

  • Limitations:
    • The current system does not support interpolation of annotations via tracking for coarsely annotated sequences, which is a feature present in some other tools like VATIC. This is attributed to the complexity of the scenarios.
  • Future Work:
    • The authors plan to incorporate more sensible suggestions for bounding boxes and gaze in un-annotated frames to further reduce annotation complexity.
  • They aim to add tracking capabilities to assist in the annotation process.

Summary updated

BibTeX
@inproceedings{bartoli2015watts,
  author = {Federico Bartoli and Lorenzo Seidenari and Giuseppe Lisanti and Svebor Karaman and Alberto Del Bimbo},
  title = {{WATTS: a Web Annotation Tool for Surveillance Scenarios}},
  booktitle = {ACM Multimedia},
  year = {2015},
  doi = {10.1145/2733373.2807411},
}
Figure