TIMo - A Dataset for Indoor Building Monitoring with a Time-of-Flight Camera

Pascal Schneider, Yuriy Anisimov, Raisul Islam, Bruno Mirbach, Jason R. Rambach, Frédéric Grandidier, Didier Stricker

Abstract

We present TIMo (Time-of-flight Indoor Monitoring), a dataset for video-based monitoring of indoor spaces captured using a time-of-flight (ToF) camera. The resulting depth videos feature people performing a set of different predefined actions, for which we provide detailed annotations. Person detection for people counting and anomaly detection are the two targeted applications. Most existing surveillance video datasets provide either grayscale or RGB videos. Depth information, on the other hand, is still a rarity in this class of datasets in spite of being popular and much more common in other research fields within computer vision. Our dataset addresses this gap in the landscape of surveillance video datasets. The recordings took place at two different locations with the ToF camera set up either in a top-down or a tilted perspective on the scene.

Introduction

Automated building management and surveillance systems traditionally rely on RGB or standard infrared (IR) cameras. However, these modalities present significant challenges regarding privacy and environmental robustness. RGB video often captures identifiable facial features, raising ethical and legal concerns in public spaces. Moreover, RGB systems are sensitive to lighting fluctuations and complex backgrounds.

The TIMo dataset addresses these issues by utilizing Time-of-Flight (ToF) technology. ToF sensors provide depth information that preserves privacy by capturing geometric shapes rather than high-resolution textures, while remaining robust against varying light conditions. While depth datasets exist, they often focus on action recognition or 3D reconstruction; TIMo is specifically designed for surveillance tasks like people counting and anomaly detection from standard monitoring perspectives.

Data Acquisition and Perspectives

The researchers utilized the Microsoft Azure Kinect to record 44 subjects across two indoor scenes. The cameras were positioned in two standard surveillance configurations: a top-down view for monitoring entrances and a tilted view for broad area coverage.

Physical camera setup for tilted and top-down perspectives

The dataset includes post-processed "undistorted" images remapped to a pinhole camera model, facilitating the conversion of 2D data into 3D point clouds. This geometric approach allows for "natural background removal," where moving objects are easily isolated from static structures like floors and walls based on their 3D position.

The TIMo Dataset

TIMo is divided into two primary subsets tailored for specific computer vision tasks:

  1. TIMo Anomaly Detection: Comprising 1,588 sequences (approx. 612,000 frames), this subset includes normal activities (e.g., walking, talking) and anomalous events such as falls, abandoned objects, or aggressive behavior. The training split contains only "normal" sequences to support unsupervised learning research.
  2. TIMo Person Detection: This subset contains 243 sequences (23,600 frames) with detailed 2D and 3D bounding boxes and segmentation masks, optimized for people counting and localization.
Sample frames showing IR and depth modalities across both perspectives

Key Results and Advantages

The dataset's scale and annotation quality set it apart from existing benchmarks. For the person detection task, the authors provided 27,687 total annotations, including 3D bounding boxes defined in the world coordinate system. By using predefined "choreographies," the authors ensured a balanced dataset with sufficient anomalous samples to train and test deep learning models effectively.

Furthermore, the choice of modern hardware (Azure Kinect) ensures the dataset reflects current-generation sensor capabilities, offering higher resolution, a wider field of view, and lower distance errors compared to older sensors like the Kinect v2.

Conclusion

TIMo provides a massive, high-quality resource for the development of privacy-preserving indoor monitoring systems. By focusing on the depth and IR modalities, it offers a robust alternative to traditional RGB-based surveillance. The dataset is publicly available for research purposes at https://vizta-tof.kl.dfki.de/timo-dataset-overview/.

Summary updated

Abstract

We present TIMo (Time-of-flight Indoor Monitoring), a dataset for video-based monitoring of indoor spaces captured using a time-of-flight (ToF) camera. The resulting depth videos feature people performing a set of different predefined actions, for which we provide detailed annotations. Person detection for people counting and anomaly detection are the two targeted applications. Most existing surveillance video datasets provide either grayscale or RGB videos. Depth information, on the other hand, is still a rarity in this class of datasets in spite of being popular and much more common in other research fields within computer vision. Our dataset addresses this gap in the landscape of surveillance video datasets. The recordings took place at two different locations with the ToF camera set up either in a top-down or a tilted perspective on the scene. The dataset is publicly available at https://vizta-tof.kl.dfki.de/timo-dataset-overview/.
BibTeX
@inproceedings{schneider2022timo,
  author = {Pascal Schneider and Yuriy Anisimov and Raisul Islam and Bruno Mirbach and Jason R. Rambach and Frédéric Grandidier and Didier Stricker},
  title = {{TIMo - A Dataset for Indoor Building Monitoring with a Time-of-Flight Camera}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=nNbFr-RZvg},
}
Figure