Abstract
We present TICaM, a Time-of-flight In-car Cabin Monitoring dataset for vehicle interior monitoring using a single wide-angle depth camera. Our dataset addresses the deficiencies of currently available in-car cabin datasets in terms of the ambit of labeled classes, recorded scenarios and provided annotations; all at the same time. We record an exhaustive list of actions performed while driving and provide for them multi-modal labeled images (depth, RGB and IR), with complete annotations for 2D and 3D object detection, instance and semantic segmentation as well as activity annotations for RGB frames. Additional to real recordings, we provide a synthetic dataset of in-car cabin images with same multi-modality of images and annotations, providing a unique and extremely beneficial combination of synthetic and real data for effectively training cabin monitoring systems and evaluating domain adaptation approaches.
Introduction
As the automotive industry progresses toward autonomous and highly automated vehicles, monitoring the interior cabin has become as critical as monitoring the external environment. Robust cabin monitoring systems are essential for occupant safety—such as optimizing airbag deployment based on the presence of an adult versus an infant in a rear-facing seat—and for driver activity recognition to ensure safe transitions between automated and manual control.
Existing datasets often fall short by relying solely on RGB video, which presents privacy concerns and performs poorly in low-light conditions. Furthermore, many focus exclusively on the driver, neglecting passengers or safety-critical objects like child seats. TICaM (Time-of-flight In-car Cabin Monitoring) addresses these gaps by providing a high-quality, multi-modal dataset specifically designed for these complex real-world environments.
The TICaM Dataset
TICaM is a comprehensive benchmark featuring 6.7K real-world and 3.3K synthetic depth images. The dataset is unique in its multi-modality, providing Depth (Z-image), Infrared (IR) amplitude, and RGB video streams simultaneously.
Data Collection Methodology
The researchers employed a dual-pronged approach to data collection:
- Real-World Recording: A realistic in-car cabin mock-up was equipped with a wide-angle Kinect Azure camera mounted near the rear-view mirror. This setup captures a production-realistic viewpoint of both the driver and passenger. Data was recorded from 13 participants performing 20 different actions across eight distinct scenarios.
- Synthetic Generation: To facilitate domain adaptation research, the team used Blender to model a Mercedes A-Class interior. They utilized human models with randomized poses, clothing, and objects to generate thousands of frames with perfect, automated ground-truth annotations.
Key Insights and Innovations
TICaM introduces several critical advancements for the field:
- Privacy-Preserving Modality: By emphasizing Time-of-Flight (ToF) depth data, the dataset enables monitoring systems that do not rely on identifiable RGB imagery, addressing consumer privacy concerns.
- Safety-Critical Scenarios: The authors explicitly included human dolls and various child seat configurations (Forward and Rearward Facing). This addresses a vital gap in existing datasets, focusing on the scenarios where computer vision is most needed for life-saving hardware adjustments.
- Synthetic-to-Real Bridge: By providing paired synthetic and real data of the same environment, TICaM serves as a primary resource for Domain Adaptation research, allowing models to be trained on synthetic data and evaluated on real-world complexity.
Results and Utility
The dataset is proven suitable for four distinct tasks: 2D object detection, 3D object detection, instance/semantic segmentation, and activity recognition. It provides 15 object classes for real data (including Infant, Child, Laptop, and Handbag) and 20 activity classes (including using phone, reaching backwards, and talking).
Conclusion
TICaM represents a significant step forward in vehicle interior monitoring. By combining multi-modal real-world data with high-fidelity synthetic data, it provides a robust foundation for developing next-generation safety systems. Future work will focus on expanding synthetic object diversity and evaluating domain adaptation algorithms to bridge the gap between simulated and real-world cabin environments.
Summary updated