X-MAS: Extremely Large-Scale Multi-Modal Sensor Dataset for Outdoor Surveillance in Real Environments

Dong-Ki Noh, Changki Sung, Teayoung Uhm, Wooju Lee, Hyungtae Lim, Jaeseok Choi, Kyuewang Lee, Dasol Hong, Daeho Um, Inseop Chung, Hochul Shin, Min-Jung Kim, Hyoung-Rock Kim, SeungMin Baek, Hyun Myung

Abstract

The X-MAS (eXtremely large-scale Multi-modAl Sensor) dataset is a massive new resource designed for outdoor robot surveillance. It features over 500,000 image pairs (2.5 million total images) across RGB, thermal, IR, depth, and LiDAR sensors. By providing high-quality, first-person view (FPV) data captured in harsh weather and low-light conditions, X-MAS bridges the gap between standard computer vision and the practical requirements of field robotics.

Introduction

While deep learning has revolutionized surveillance tasks like human detection and action recognition, existing datasets often suffer from a "perspective problem." Most are captured from static, high-angle CCTV cameras or are tailored for autonomous driving. When models trained on these datasets are deployed on mobile surveillance robots, performance degrades because the robot's first-person perspective is fundamentally different.

Furthermore, standard RGB cameras fail in extreme outdoor conditions such as heavy rain, thick fog, or total darkness. To build truly robust surveillance robots, researchers need data that combines multiple sensing modalities with the specific viewpoint and environmental challenges a mobile robot faces.

Extreme environmental conditions like thick fog or smoke present significant challenges for standard RGB sensors, necessitating multi-modal approaches.

The X-MAS Dataset

The primary contribution of this work is the X-MAS dataset, which provides a unique combination of FPV, multi-modality, and high environmental diversity. Key features include:

  • Scale: Over 2.5 million images across 2,624 sequences.
  • Modalities: RGB, Thermal, IR (Night Vision), Depth, and 3D LiDAR scans.
  • Annotations: Comprehensive labels including bounding boxes, pixel-level masks, tracking IDs, and action labels.
  • Diversity: Data was captured across various weather conditions and times of day. Professional actors performed "normal" and "anomalous" actions, such as falling, running, or carrying props, to simulate real-world security scenarios.
The dataset includes specific "anomalous" actions, such as a person lying on the ground, captured from a robot's first-person perspective at night.

Methodology and Key Insights

The researchers developed a custom multi-modal sensor module deployed on both fixed platforms and mobile robots. Technical contributions include rigorous sensor synchronization and calibration across all five modalities.

Key insights from the study include:

  • Thermal is Essential: Thermal and IR data are critical for maintaining surveillance performance at night or in fog where RGB sensors become useless.
  • The "Robot View" Matters: Training on FPV data is vital for mobile robots. The dataset addresses robot-centric challenges like motion blur and varying sensor heights.
  • Complementary Modalities: By combining 2D detections from RGB/Thermal with 1D depth data from LiDAR, the team implemented a robust 3D Kalman filter for tracking that outperforms 2D-only methods.

Experimental Results

The authors validated the dataset using several state-of-the-art models:

  • Object Detection: Fine-tuning YOLOv5 on X-MAS + COCO significantly improved performance. For thermal images at night, the mAP jumped from a baseline of 5.5% (COCO-only) to 19.6% when using X-MAS data.
  • Action Classification: Using a modified ResNet-18, they achieved 95.6% accuracy for daytime (RGB) and 93.4% accuracy for nighttime (Thermal) action recognition.
  • Tracking: Their multi-modal tracker achieved a Multiple Object Tracking Accuracy (MOTA) of 66.8% at night using thermal data, demonstrating real-time capability (>30Hz) on embedded hardware like the Jetson AGX Xavier.

Conclusion

The X-MAS dataset provides a critical resource for developing robust outdoor surveillance robots. While the current suite is extensive, future work will focus on more efficient image fusion techniques to handle sensors with different Fields of View (FOVs) and expanding the dataset to even more diverse global environments beyond the initial test sites.

Summary updated

Abstract

In robotics and computer vision communities, extensive studies have been widely conducted regarding surveillance tasks, including human detection, tracking, and motion recognition with a camera. Additionally, deep learning algorithms are widely utilized in the aforementioned tasks as in other computer vision tasks. Existing public datasets are insufficient to develop learning-based methods that handle various surveillance for outdoor and extreme situations such as harsh weather and low illuminance conditions. Therefore, we introduce a new large-scale outdoor surveillance dataset named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e</i> <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</b> <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">tremely large-scale</i> <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</b> <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ulti-mod</i> <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</b> <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</b> <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ensor dataset</i> ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X-MAS</i> ) containing more than 500,000 image pairs and the first-person view data annotated by well-trained annotators. Moreover, a single pair contains multi-modal data (e.g. an IR image, an RGB image, a thermal image, a depth image, and a LiDAR scan). This is the first large-scale first-person view outdoor multi-modal dataset focusing on surveillance tasks to the best of our knowledge. We present an overview of the proposed dataset with statistics and present methods of exploiting our dataset with deep learning-based algorithms.
BibTeX
@article{noh2023x,
  author = {Dong-Ki Noh and Changki Sung and Teayoung Uhm and Wooju Lee and Hyungtae Lim and Jaeseok Choi and Kyuewang Lee and Dasol Hong and Daeho Um and Inseop Chung and Hochul Shin and Min-Jung Kim and Hyoung-Rock Kim and SeungMin Baek and Hyun Myung},
  title = {{X-MAS: Extremely Large-Scale Multi-Modal Sensor Dataset for Outdoor Surveillance in Real Environments}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2023},
  doi = {10.1109/lra.2023.3236569},
}
Figure