TIMo - A Dataset for Indoor Building Monitoring with a Time-of-Flight Camera

Pascal Schneider, Yuriy Anisimov, Raisul Islam, Bruno Mirbach, Jason R. Rambach, Didier Stricker, Frédéric Grandidier

TL;DR

The TIMo (Time-of-flight Indoor Monitoring) dataset provides over 600,000 frames of depth and infrared data specifically designed for indoor surveillance tasks like people counting and anomaly detection. By leveraging modern Time-of-Flight (ToF) technology, the authors offer a high-resolution, privacy-preserving alternative to traditional RGB video datasets.

What Problem Does This Paper Address?

Traditional automated building management and surveillance systems rely heavily on RGB or standard infrared (IR) cameras. However, these modalities face two major hurdles: privacy and robustness. RGB video makes individuals easily identifiable, raising significant ethical and legal concerns in public or office spaces. Furthermore, RGB systems often struggle with changing lighting conditions and complex backgrounds.

While depth-sensing technology (like the Microsoft Kinect) has been around for a decade, most existing depth datasets focus on action recognition (e.g., "how to drink water") or static 3D scene reconstruction. There is a distinct lack of large-scale depth data captured from the specific perspectives used in surveillance—namely, top-down and tilted-view angles—aimed at detecting anomalies like falls or suspicious behavior.

Methods

The researchers used the Microsoft Azure Kinect, a modern ToF camera, to record 44 different subjects across two distinct indoor scenes. To ensure the data is useful for real-world applications, they set up the cameras in two standard surveillance configurations:

  1. Top-down View: Mounted directly above entrances to monitor flow.
  2. Tilted View: Mounted at an angle to cover a wide area of a room.

The team developed a custom software tool to manually annotate the data in 3D. They also applied post-processing to provide "undistorted" images remapped to a common pinhole camera model, making it easier for researchers to convert the 2D images into 3D point clouds.

Datasets

The paper introduces the TIMo dataset, which is split into two primary subsets:

  • TIMo Anomaly Detection: Contains 1,588 sequences (612,000 frames). It includes "normal" activities (walking, talking, getting coffee) and "anomalous" events (leaving objects behind, arguing, throwing things, or a person collapsing).
  • TIMo Person Detection: Contains 243 sequences (23,600 frames) with detailed 2D and 3D bounding boxes and segmentation masks for people counting and localization.

The authors compare TIMo to existing benchmarks like ShanghaiTech (RGB only) and NTU RGB+D (action-focused), highlighting that TIMo is one of the few datasets to provide high-resolution ToF data specifically for indoor monitoring.

Key Insights and Ideas

  • Privacy-by-Design: Depth data provides enough geometric information to detect a person or an event (like a fall) without capturing the high-resolution facial features or textures that make a person identifiable in RGB.
  • Geometric Advantages: Unlike RGB, depth data allows for "natural geometrical background removal." Since the distance to the floor and walls is known, the system can easily isolate moving objects based on their 3D position.
  • Controlled Anomalies: Rather than waiting for rare real-world events, the authors used predefined "choreographies." This allowed them to create a balanced dataset with enough anomalous samples to actually train and test deep learning models effectively.
  • Modern Hardware Matters: By using the Azure Kinect instead of the older Kinect v2, the dataset offers higher resolution, a wider field of view, and lower distance errors, reflecting the capabilities of current-generation sensors.

Key Results

  • Scale: The dataset is massive, featuring over 612,000 individual video frames.
  • Rich Annotations: For the person detection task, they provided 27,687 total annotations, including 3D bounding boxes defined by a box center $(cx, cy, cz)$ and dimensions $(dx, dy, dz)$ in the world coordinate system.
  • Unsupervised Ready: The training split for the anomaly dataset contains only normal sequences. This is specifically designed to support the development of unsupervised learning algorithms that learn what "normal" looks like and flag anything else as an outlier.

Limitations and Future Work

The authors acknowledge a few limitations:

  • Background Variety: To maintain high-quality calibration and avoid "learning the background" instead of the action, they limited the recordings to a few specific scenes.
  • Modality Focus: While the camera can record RGB, the authors intentionally excluded it from the public dataset to focus on the depth/IR modality and maintain the privacy-centric nature of the project.
  • Future Directions: They suggest that as ToF sensors continue to improve in resolution and sensitivity, combining this data with even more sophisticated deep-learning architectures will enable complex behavior analysis beyond simple counting or fall detection.

Check out the dataset: The TIMo dataset is publicly available for research purposes at https://vizta-tof.kl.dfki.de/timo-dataset-overview/.

Summary updated

Abstract

We present TIMo (Time-of-flight Indoor Monitoring), a dataset for video-based monitoring of indoor spaces captured using a time-of-flight (ToF) camera. The resulting depth videos feature people performing a set of different predefined actions, for which we provide detailed annotations. Person detection for people counting and anomaly detection are the two targeted applications. Most existing surveillance video datasets provide either grayscale or RGB videos. Depth information, on the other hand, is still a rarity in this class of datasets in spite of being popular and much more common in other research fields within computer vision. Our dataset addresses this gap in the landscape of surveillance video datasets. The recordings took place at two different locations with the ToF camera set up either in a top-down or a tilted perspective on the scene. Moreover, we provide experimental evaluation results from baseline algorithms.
BibTeX
@inproceedings{schneider2022timo,
  author = {Pascal Schneider and Yuriy Anisimov and Raisul Islam and Bruno Mirbach and Jason R. Rambach and Didier Stricker and Frédéric Grandidier},
  title = {{TIMo - A Dataset for Indoor Building Monitoring with a Time-of-Flight Camera}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=nNbFr-RZvg},
}

Loading related content...

Figure