Abstract
The X-MAS (eXtremely large-scale Multi-modAl Sensor) dataset is a massive new resource designed for outdoor robot surveillance. It features over 500,000 image pairs (2.5 million total images) across RGB, thermal, IR, depth, and LiDAR sensors. By providing high-quality, first-person view (FPV) data captured in harsh weather and low-light conditions, X-MAS bridges the gap between standard computer vision and the practical requirements of field robotics.
Introduction
While deep learning has revolutionized surveillance tasks like human detection and action recognition, existing datasets often suffer from a "perspective problem." Most are captured from static, high-angle CCTV cameras or are tailored for autonomous driving. When models trained on these datasets are deployed on mobile surveillance robots, performance degrades because the robot's first-person perspective is fundamentally different.
Furthermore, standard RGB cameras fail in extreme outdoor conditions such as heavy rain, thick fog, or total darkness. To build truly robust surveillance robots, researchers need data that combines multiple sensing modalities with the specific viewpoint and environmental challenges a mobile robot faces.
The X-MAS Dataset
The primary contribution of this work is the X-MAS dataset, which provides a unique combination of FPV, multi-modality, and high environmental diversity. Key features include:
- Scale: Over 2.5 million images across 2,624 sequences.
- Modalities: RGB, Thermal, IR (Night Vision), Depth, and 3D LiDAR scans.
- Annotations: Comprehensive labels including bounding boxes, pixel-level masks, tracking IDs, and action labels.
- Diversity: Data was captured across various weather conditions and times of day. Professional actors performed "normal" and "anomalous" actions, such as falling, running, or carrying props, to simulate real-world security scenarios.
Methodology and Key Insights
The researchers developed a custom multi-modal sensor module deployed on both fixed platforms and mobile robots. Technical contributions include rigorous sensor synchronization and calibration across all five modalities.
Key insights from the study include:
- Thermal is Essential: Thermal and IR data are critical for maintaining surveillance performance at night or in fog where RGB sensors become useless.
- The "Robot View" Matters: Training on FPV data is vital for mobile robots. The dataset addresses robot-centric challenges like motion blur and varying sensor heights.
- Complementary Modalities: By combining 2D detections from RGB/Thermal with 1D depth data from LiDAR, the team implemented a robust 3D Kalman filter for tracking that outperforms 2D-only methods.
Experimental Results
The authors validated the dataset using several state-of-the-art models:
- Object Detection: Fine-tuning YOLOv5 on X-MAS + COCO significantly improved performance. For thermal images at night, the mAP jumped from a baseline of 5.5% (COCO-only) to 19.6% when using X-MAS data.
- Action Classification: Using a modified ResNet-18, they achieved 95.6% accuracy for daytime (RGB) and 93.4% accuracy for nighttime (Thermal) action recognition.
- Tracking: Their multi-modal tracker achieved a Multiple Object Tracking Accuracy (MOTA) of 66.8% at night using thermal data, demonstrating real-time capability (>30Hz) on embedded hardware like the Jetson AGX Xavier.
Conclusion
The X-MAS dataset provides a critical resource for developing robust outdoor surveillance robots. While the current suite is extensive, future work will focus on more efficient image fusion techniques to handle sensors with different Fields of View (FOVs) and expanding the dataset to even more diverse global environments beyond the initial test sites.
Summary updated