TICaM: A Time-of-flight In-car Cabin Monitoring Dataset

Jigyasa Singh Katrolia, Ahmed El-Sherif, Hartmut Feld, Bruno Mirbach, Jason R. Rambach, Didier Stricker

TL;DR

TICaM is a comprehensive, multi-modal dataset designed to advance in-car cabin monitoring using Time-of-Flight (ToF) technology. By providing 6.7K real-world and 3.3K synthetic depth images—complete with 2D/3D bounding boxes, segmentation masks, and activity labels—it addresses critical gaps in occupant safety and driver activity recognition.

What Problem Does This Paper Address?

As we move toward autonomous and highly automated vehicles, monitoring the interior cabin becomes as important as monitoring the road. Current systems need to understand who is in the car (occupant classification), what they are doing (activity recognition), and where they are positioned to ensure safety—for instance, adjusting airbag deployment based on whether a seat holds an adult, a child in a rear-facing seat, or an inanimate object.

However, existing datasets often fall short. Many rely solely on RGB video (which raises privacy concerns and struggles with lighting), focus only on the driver, or lack the diverse scenarios found in everyday life, such as passengers, infants in car seats, or various handheld objects. TICaM (Time-of-flight In-car Cabin Monitoring) aims to be a "one-stop-shop" dataset that provides high-quality, multi-modal data for these complex real-world environments.

Methods

The researchers at DFKI developed a dual-pronged approach to data collection:

  1. Real-World Recording: They built a realistic in-car cabin mock-up equipped with a wide-angle Kinect Azure camera mounted near the rear-view mirror. This setup captures a "production-realistic" viewpoint of both the driver and passenger. They recorded 13 participants performing 20 different actions across eight distinct scenarios (e.g., driver alone, driver with an infant in a rear-facing seat, driver with a passenger).
  2. Synthetic Generation: To augment the real data and facilitate domain adaptation research, they used Blender to create a synthetic version of the dataset. They modeled a Mercedes A-Class interior and used human models with randomized poses, clothing, and objects to simulate thousands of additional frames with perfect, automated ground-truth annotations.

The dataset is unique in its multi-modality, providing Depth (Z-image), Infrared (IR) amplitude, and RGB video streams simultaneously.

Datasets

TICaM introduces a new benchmark and compares it against several existing datasets:

  • TICaM (Ours): 6.7K real depth/IR images, 123K RGB frames, and 3.3K synthetic images. Includes 2D/3D boxes, masks, and 20 activity classes.
  • SVIRO: A synthetic-only dataset focused on the rear seat.
  • Drive&Act: A large-scale dataset with many views and activities, but lacking the specific 3D depth/IR focus and occupant variety of TICaM.
  • AUC / Brain4Cars / HEH: Older or more limited datasets focusing primarily on driver distraction or specific maneuvers.

Key Insights and Ideas

  • Privacy-Preserving Modality: By emphasizing Time-of-Flight (ToF) depth data, the researchers highlight a path toward cabin monitoring that doesn't rely on identifiable RGB imagery, making it more palatable for consumer privacy.
  • The "Safety-Critical" Gap: The authors explicitly included human dolls and various child seat configurations (Forward Facing and Rearward Facing). This is a vital "aha" moment: most datasets ignore the very scenarios where computer vision is needed most for life-saving hardware adjustments (like airbag suppression).
  • Synthetic-to-Real Bridge: By providing both synthetic and real data of the same environment, TICaM serves as a playground for Domain Adaptation. Researchers can train models on "cheap" synthetic data and use TICaM to evaluate how well those models generalize to the messy reality of a physical car cabin.

Key Results

  • Comprehensive Annotations: The dataset provides 15 object classes for real data (including Infant, Child, Laptop, Handbag) and 20 activity classes (including using phone, reaching backwards, talking).
  • Data Volume:
    • Activity Recognition: 86K training frames / 37K testing frames.
  • Object Detection/Segmentation: 4.7K real training images / 2.0K testing images, plus 3.3K synthetic training images.
  • Multi-Purpose Utility: The dataset is proven suitable for four distinct tasks: 2D object detection, 3D object detection, instance/semantic segmentation, and activity recognition.

Limitations and Future Work

While TICaM is a significant step forward, the authors note some limitations:

  • Synthetic Diversity: The synthetic dataset currently features fewer object classes (4) compared to the real-world dataset (15). Expanding the synthetic library to match the real-world complexity would further improve domain adaptation.
  • Static Mock-up: The real-world data was recorded in a simulator/mock-up rather than a moving vehicle on a bumpy road, which might introduce different noise patterns in the ToF sensor.
  • Future Directions: The authors suggest that the next logical step is using this dataset to benchmark state-of-the-art domain adaptation algorithms, specifically seeing how well a model trained on the synthetic cabin performs on the real-world participants.

Check out the dataset: https://vizta-tof.kl.dfki.de/

Summary updated

Abstract

We present TICaM, a Time-of-flight In-car Cabin Monitoring dataset for vehicle interior monitoring using a single wide-angle depth camera. Our dataset addresses the deficiencies of currently available in-car cabin datasets in terms of the ambit of labeled classes, recorded scenarios and provided annotations; all at the same time. We record an exhaustive list of actions performed while driving and provide for them multi-modal labeled images (depth, RGB and IR), with complete annotations for 2D and 3D object detection, instance and semantic segmentation as well as activity annotations for RGB frames. Additional to real recordings, we provide a synthetic dataset of in-car cabin images with same multi-modality of images and annotations, providing a unique and extremely beneficial combination of synthetic and real data for effectively training cabin monitoring systems and evaluating domain adaptation approaches. The dataset is available at https://vizta-tof.kl.dfki.de/.
BibTeX
@inproceedings{katrolia2021ticam,
  author = {Jigyasa Singh Katrolia and Ahmed El-Sherif and Hartmut Feld and Bruno Mirbach and Jason R. Rambach and Didier Stricker},
  title = {{TICaM: A Time-of-flight In-car Cabin Monitoring Dataset}},
  booktitle = {BMVC},
  year = {2021},
  url = {https://openreview.net/forum?id=zLjXjRjXlZm},
}

Loading related content...

Figure