Standard Datasets for Autonomous Navigation and Mapping: A Full-Stack Construction Methodology

Yuanzhi Liu, Yujia Fu, Minghui Qin, Yufeng Xu, Bin Cui, Kunhua Liu, Fengdong Chen, Wei Tao, Michiel Vlaminck, Bart Goossens, Poly Z. H. Sun, Hui ZhaoIEEE Transactions on Intelligent Vehicles 2024 · DOI

Abstract

The development of intelligent Vehicles (IVs) requires extensive standard datasets for training, benchmarking, and improvement. Autonomous Navigation and Mapping (ANM), as a critical technology for IVs, imposes exceptionally high demands on dataset construction. This is significant in its requirements for comprehensive sensor calibration, precise time synchronization, and accurate generation of ground truth. Besides, the whole construction workflow also demands intricate knowledge and sophisticated practices, necessitating lengthy learning curves for researchers to attain proficiency. The above challenges have led to a slow production of qualified datasets, directly constraining the advancement of ANM. However, so far, an investigation focused on a mature construction methodology of ANM dataset is still missing. This paper strives to fill the gap. Specifically, based on our systematic reviews and extensive practices, for the first time, a full-stack construction methodology of ANM dataset is proposed, including modules of platform construction, sensor calibration, time synchronization, ground truth generation, synthetic data production, and benchmark criteria, with detailed techniques and methodological routes provided in each step. Several long-standing issues are resolved within the methodology. Importantly, we introduce versatile calibration and synchronization frameworks that attain up to us-level and mm-level precision. Besides, we propose a full-scenario ground truth system that can generate scene-map and trajectory at cm-level accuracy. To verify the effectiveness of our methodology, we design a high-quality dataset and benchmark multiple state-of-the-art algorithms on it. The successful workflow demonstrates that our methodology can significantly reduce the research threshold and help individuals and institutions to construct datasets in a standardized way.

Introduction

Creating a "gold standard" dataset for autonomous vehicles is deceptively difficult. While many researchers rely on established datasets like KITTI or EuRoC, these often harbor hidden technical limitations. For instance, software-induced temporal errors—sometimes as high as 10ms—can significantly degrade the performance of high-speed SLAM (Simultaneous Localization and Mapping) algorithms. Furthermore, the lack of a standardized "instruction manual" for sensor fusion, hardware triggering, and ground truth generation has created a high barrier to entry, leading to a shortage of diverse data for challenging "corner cases" like extreme weather or indoor-outdoor transitions.

A Full-Stack Methodology

To address these challenges, we propose a modular, five-phase construction pipeline designed to standardize the production of high-quality ANM datasets. This methodology moves dataset production from an "artisan" craft to a repeatable, standardized process.

The five-phase pipeline for standardized ANM dataset construction.

The pipeline encompasses:

  1. Platform Construction: Designing a robust vehicle with a comprehensive sensor suite.
  2. Spatial-Temporal Alignment: Achieving precise extrinsic calibration and microsecond-level synchronization.
  3. Data Collection: Implementing rigorous policies for diverse environmental scenarios.
  4. Ground Truth Generation: Producing high-accuracy reference trajectories and maps.
  5. Benchmark Criteria: Establishing standardized metrics for evaluation.

Platform and Sensor Integration

A key component of our methodology is the "full-modal" sensor suite. We integrated stereo cameras, spinning and MEMS LiDAR, industrial-grade IMUs, and GNSS onto a four-wheeled robotic platform.

The 'full-modal' sensor suite and robot platform used to validate the methodology.

This hardware setup allows for the collection of rich, redundant data across various modalities. By open-sourcing these hardware selection and integration guidelines, we aim to lower the research threshold for the broader community.

Technical Breakthroughs

Precision Time Synchronization

We moved beyond traditional software-based synchronization (NTP) to a multi-host Precision Time Protocol (PTP) architecture. By utilizing hardware triggers and PTP-synchronized host machines, we achieved a synchronization precision of 1.56 microseconds (µs), effectively eliminating the temporal jitter common in older datasets.

Ground Truth via MapLoc

Generating ground truth in GNSS-denied environments, such as underground garages, is a major hurdle. Our "MapLoc" system resolves this by using survey-grade laser scanners (e.g., Leica RTC360) to create a high-precision 3D prior map. The robot then localizes within this "gold standard" map using its onboard sensors, yielding trajectory ground truth with centimeter-level accuracy (RMSE of ~11mm indoors and ~8mm outdoors).

The ParkingLot Dataset and Benchmarking

To validate our methodology, we introduced the ParkingLot Dataset, comprising 8 sequences over 8.27 km. This dataset specifically targets challenging scenarios such as indoor-outdoor transitions, urban canyons, and lush vegetation.

Benchmarking state-of-the-art (SOTA) algorithms like VINS-Mono, ORB-SLAM3, and LVI-SAM on this dataset revealed significant performance gaps in complex environments. These results underscore the dataset's value as a rigorous benchmark for modern autonomous navigation systems, highlighting that even top-tier algorithms struggle with dense vegetation and lighting transitions.

Conclusion and Future Work

Our full-stack methodology provides a comprehensive blueprint for constructing high-quality ANM datasets. While the current approach requires specialized hardware, future work will focus on reducing costs and developing Automatic Data Annotation Systems to further streamline the data production cycle and bridge the "realism gap" in synthetic data simulation.

Summary updated

Abstract

The development of intelligent Vehicles (IVs) requires extensive standard datasets for training, benchmarking, and improvement. Autonomous Navigation and Mapping (ANM), as a critical technology for IVs, imposes exceptionally high demands on dataset construction. This is significant in its requirements for comprehensive sensor calibration, precise time synchronization, and accurate generation of ground truth. Besides, the whole construction workflow also demands intricate knowledge and sophisticated practices, necessitating lengthy learning curves for researchers to attain proficiency. The above challenges have led to a slow production of qualified datasets, directly constraining the advancement of ANM. However, so far, an investigation focused on a mature construction methodology of ANM dataset is still missing. This paper strives to fill the gap. Specifically, based on our systematic reviews and extensive practices, for the first time, a full-stack construction methodology of ANM dataset is proposed, including modules of platform construction, sensor calibration, time synchronization, ground truth generation, synthetic data production, and benchmark criteria, with detailed techniques and methodological routes provided in each step. Several long-standing issues are resolved within the methodology. Importantly, we introduce versatile calibration and synchronization frameworks that attain up to us-level and mm-level precision. Besides, we propose a full-scenario ground truth system that can generate scene-map and trajectory at cm-level accuracy. To verify the effectiveness of our methodology, we design a high-quality dataset and benchmark multiple state-of-the-art algorithms on it. The successful workflow demonstrates that our methodology can significantly reduce the research threshold and help individuals and institutions to construct datasets in a standardized way.
BibTeX
@article{liu2024standard,
  author = {Yuanzhi Liu and Yujia Fu and Minghui Qin and Yufeng Xu and Bin Cui and Kunhua Liu and Fengdong Chen and Wei Tao and Michiel Vlaminck and Bart Goossens and Poly Z. H. Sun and Hui Zhao},
  title = {{Standard Datasets for Autonomous Navigation and Mapping: A Full-Stack Construction Methodology}},
  journal = {IEEE Transactions on Intelligent Vehicles},
  year = {2024},
  doi = {10.1109/tiv.2024.3360273},
}
Figure