MS-DOTT: A Multi-Modal Dataset for Off-Road Traversability Estimation

Shivani Subramanyan, Saksham Sharma, Samahith Sa, Ahaan Kotian, Suresh SundaramarXiv 2026 · arXiv

TL;DR

MS-DOTT is a new multi-modal dataset designed to push the boundaries of off-road autonomous navigation by bridging the gap between long-range environmental context and fine-grained, near-field hazard detection. Featuring a diverse sensor suite—including HDR cameras, thermal imaging, and high-density Time-of-Flight (ToF) sensors—it provides 3,000 meters of day and night trajectories to help researchers tackle the "unstructured" challenges of off-road traversability.

What Problem Does This Paper Address?

Navigating off-road is a significantly harder problem than urban driving. In a city, you have lanes, signs, and predictable surfaces. Off-road, a patch of "vegetation" might be traversable grass or a wheel-swallowing ditch hidden by weeds. Furthermore, lighting conditions in forests or fields are extreme, ranging from blinding glare to total darkness.

The authors identify three critical gaps in existing datasets:

  1. Semantic Ambiguity: Knowing a region is "grass" isn't enough; you need to know its slope and roughness to determine if it's actually traversable.
  2. Illumination Limits: Most datasets fail in low-light or high-glare scenarios.
  3. Scale Mismatch: Sensors often focus on either long-range mapping (LiDAR) or short-range vision, but rarely provide the high-resolution, near-field data needed to spot "micro-obstacles" like small rocks or roots right in front of the vehicle.

Methods

The researchers built a custom "Data Collection Bench" (shown in Figure 2) mounted on a Botsync Copernicus UGV.

Fig. 2: MS-DOTT Data Collection Bench.

The sensor suite is impressively comprehensive:

  • Vision: Standard RGB (FLIR), HDR RGB (Lucid TDR054S) for high dynamic range, Thermal (Trimer LWIR) for night, and Stereo (ZED2).
  • 3D Sensing: Long-range solid-state LiDAR (Livox HAP), mid-range LiDAR (Livox MID360), and a high-density Time-of-Flight (ToF) camera (Lucid Helios2+) for ultra-dense near-field detail.
  • Additional: 4D RADAR (Continental ARS548) for weather-agnostic velocity sensing and an Xsens IMU/GNSS for localization.

To process this data, they developed custom ROS2 drivers and a CUDA-accelerated "point painting" algorithm to project image data onto 3D point clouds in real-time. For evaluation, they adapted SALON, a state-of-the-art traversability fusion method, swapping its standard vision backbone for AnyThermal to enable traversability estimation in total darkness.

Datasets

The primary contribution is the MS-DOTT (Multi-Sensor Dataset for Off-road Terrain Traversability).

  • Scope: 11 trajectories (6 day, 5 night) covering 3000m in forest-like environments.
  • Volume: 90 minutes of data, 432,000 camera frames, and 243,000 point clouds.
  • Comparison: Unlike predecessors like RUGD or RELLIS-3D, MS-DOTT includes ToF and 4D Radar, and specifically targets the fusion of long-range and fine-grained short-range data across day and night.

Key Insights and Ideas

  • HDR is Essential for Glare: Standard RGB cameras often lose all detail in high-contrast shadows or direct sunlight (glare). The HDR camera successfully captures obstacles that are "invisible" to standard sensors due to overexposure.
  • The ToF "Micro-Obstacle" Advantage: Traditional LiDARs can be too sparse at very close ranges. The high-density ToF sensor captures small hazards like pebbles and roots that traditional 3D sensing might miss, providing a much more expressive local costmap.
  • Thermal-LiDAR Fusion for Night Ops: By using thermal features instead of RGB, the authors show that traversability costmaps can remain accurate even when visual cameras are completely underexposed.
  • Multi-Scale Sensing: The dataset proves that a single sensor isn't enough. Effective off-road navigation requires a "hierarchy" of sensors: ToF for the immediate path, mid-range LiDAR for local mapping, and long-range LiDAR for global path planning.

Key Results

The authors qualitatively demonstrated the utility of MS-DOTT by comparing costmaps generated from different sensor combinations.

Comparison of visual sensors. HDR and Thermal (right) correctly identify obstacles and traversable paths where standard FLIR (left) fails due to lighting.

As seen in Figure 3, in low-light conditions, the standard FLIR camera falsely marks clear areas as high-cost (red), while the HDR and Thermal sensors correctly identify the traversable terrain (yellow/green).

Comparison of LiDAR sensors. The ToF sensor (top right) captures near-field obstacles that the HAP and MID360 miss.

In Figure 4, the ToF sensor identifies near-field obstacles as high-cost (green box), whereas the more traditional HAP and MID360 LiDARs fail to resolve these close-in hazards (red boxes).

Limitations and Future Work

While comprehensive, the authors acknowledge several limitations:

  • Field of View: The sensors are strictly forward-facing, lacking the 360-degree coverage found in some other recent datasets.
  • Synchronization: The dataset relies on software-based synchronization (median offset of 8.7ms) rather than hardware-level triggering.
  • Trajectory Length: Due to the massive data volume generated by nine sensors, trajectories are relatively short (~250m each), which limits their use for modeling high-speed vehicle dynamics.
  • ToF Sensitivity: The ToF sensor, while excellent at night, sees a significant reduction in point cloud density under harsh direct sunlight.

Future work aims to address these by expanding the geographic diversity of the testing sites and implementing hardware-level time synchronization.

Summary updated

Abstract

Autonomous robots are gaining traction in off-road environments, driven by their rising demand in agriculture, defense, and search-and-rescue operations. However, these environments are inherently unpredictable and navigation remains a formidable challenge given the complex environmental conditions including challenging terrain and varying illumination. Off-road autonomy requires robust perception across multiple spatial scales and imaging modalities, yet most sensing setups struggle to capture both far-field environmental context and near-field terrain detail simultaneously across various illumination conditions. To facilitate development of robust off-road perception and traversability estimation systems, we propose MS-DOTT (Multi-Sensor Dataset for Off-road Terrain Traversability) a comprehensive multi-modal off-road dataset with long-range 3D sensing and short-range, fine-grained perception to facilitate high-fidelity terrain understanding by combining proprioceptive signals for tracking vehicle dynamics with exteroceptive imaging and pointcloud data. The dataset includes trajectories recorded across nominal lighting, low-light, and no-light conditions with diverse 2D and 3D sensing for both long-range large-scale and short-range high-resolution sensing, as well as 4D RADAR measurements that provide velocity estimates for detected pointclouds. We demonstrate the utility of the dataset by adapting a SOTA multi-sensor fusion method for the purpose of traversability estimation. This dataset will aid the robotics community in advancing research and development of methods for traversability estimation, off-road perception and multi-sensor fusion in unstructured environments.
BibTeX
@misc{subramanyan2026ms,
  author = {Shivani Subramanyan and Saksham Sharma and Samahith SA and Ahaan Kotian and Suresh Sundaram},
  title = {{MS-DOTT: A Multi-Modal Dataset for Off-Road Traversability Estimation}},
  year = {2026},
  eprint = {2611.02992},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02992},
}
Figure