Seamless Virtual Reality With Integrated Synchronizer and Synthesizer for Autonomous Driving

He Li, Ruihua Han, Zirui Zhao, Wei Xu, Qi Hao, Shuai Wang, Cheng-Zhong XuIEEE RA-L 2024 · DOI · OpenReview

Abstract

This paper introduces Seamless Virtual Reality (SVR), a novel framework for training autonomous driving (AD) systems by blending the safety of simulation with the physical complexity of the real world. By employing an integrated synchronizer and synthesizer ($IS^2$), SVR achieves cm-level alignment between real vehicles and virtual obstacles. Experimental results demonstrate that SVR-trained models significantly outperform those trained on pure simulation, particularly in safety-critical edge cases, achieving an 87% collision avoidance success rate in adversarial scenarios.

Introduction

Training autonomous vehicles (AVs) requires massive amounts of data, especially for "boundary cases" like near-misses, cut-ins, or crashes. Collecting this data in the real world is dangerous and expensive, while pure simulation often suffers from a "sim-to-real gap"—where the AI learns quirks of the simulator that don't exist in reality.

Virtual Reality Autonomous Driving (VRAD) attempts to bridge this by placing a real vehicle in a "symbiotic" world where it interacts with virtual obstacles. However, existing VRAD systems suffer from VR inconsistency. If the low-level data collection (where the car thinks it is) isn't perfectly synced with the high-level image synthesis (where the virtual obstacle appears to be), the resulting data is ineffective. This leads to "model clipping" (objects overlapping) or "missed collisions," rendering the training data unsuitable for real-world deployment.

Methods: The $IS^2$ Framework

The core innovation is the Integrated Synchronizer and Synthesizer ($IS^2$) design, which consists of two main modules:

  1. Drift-Aware Lidar-Inertial Synchronizer (LIS): Unlike standard localization, LIS uses a specialized iterated Kalman filter that specifically minimizes the drift between the virtual and real worlds. It operates at 50Hz, achieving cm-level accuracy (averaging 3 cm error) even in GPS-denied environments.
  2. Motion-Aware Deep Visual Synthesis Network (DVSN): This module handles the augmented reality (AR) synthesis. It utilizes a neighborhood consensus network (NC-Net) combined with a Motion-Aware Adaptive Filter (MAAF). By predicting the vehicle's trajectory using an Ackerman motion model, it focuses on a specific "Region of Interest" to align virtual and real images. Furthermore, it employs the Segment Anything Model (SAM) and YOLOv5 to cleanly extract virtual objects and integrate them into real-world camera feeds without visual artifacts.
The Seamless Virtual Reality (SVR) framework overview, featuring the integrated synchronizer and synthesizer ($IS^2$) pipeline.

The system was implemented using ROS (Robot Operating System) and CARLA, connecting a physical "LIMO" robot to a virtual environment. The DVSN module produces high-fidelity AR images that serve as the training input for the neural network.

High-fidelity AR image generation blending a virtual vehicle into a real-world scene.

Key Insights and Results

The researchers evaluated SVR using a dataset of 14,000 samples to train a conditional Imitation Learning (IL) network. Key findings include:

  • Visual Fidelity: The DVSN achieved a 98.8% recognizable rate for objects and a tiny 3.2% image deviation, significantly outperforming standard matching algorithms.
  • Driving Performance: In adversarial tests (cut-ins and red-light runners), the SVR-trained model achieved an 87% collision avoidance success rate, compared to just 23% for models trained on purely virtual data. It also reduced the need for human intervention by 10x compared to the virtual-only baseline.
  • Knowledge Transfer: A key finding was that the SVR-trained model could handle adversarial scenarios in the real world, even if it had only "seen" those specific hazards in the virtual portion of the SVR environment.

Conclusion

SVR provides a high-fidelity, safe, and scalable solution for training AD systems on complex edge cases. While current performance depends on initial offline calibration, the framework demonstrates the power of near-perfect spatial synchronization in bridging the sim-to-real gap. Future work aims to integrate Large Multi-modal Models (LMMs) for more diverse environmental variations and to test the system in higher-speed, complex urban environments.

Summary updated

Abstract

Virtual reality (VR) is a promising data engine for autonomous driving (AD). However, data fidelity in this paradigm is often degraded by VR inconsistency, for which the existing VR approaches become ineffective, as they ignore the inter-dependency between low-level VR synchronizer designs (i.e., data collector) and high-level VR synthesizer designs (i.e., data processor). This paper presents a seamless virtual reality ($\mathsf {SVR}$) platform for AD, which mitigates such inconsistency, enabling VR agents to interact with each other in a shared symbiotic world. The crux to $\mathsf {SVR}$ is an integrated synchronizer and synthesizer ($\mathsf {IS}^{2}$) design, which consists of a drift-aware lidar-inertial synchronizer for VR colocation and a motion-aware deep visual synthesis network for augmented reality image generation. We implement $\mathsf {SVR}$ on car-like robots in two sandbox platforms, achieving a cm-level VR colocalization accuracy and 3.2% VR image deviation, thereby avoiding missed collisions or model clippings. Experiments show that the proposed $\mathsf {SVR}$ reduces the intervention times, missed turns, and failure rates compared to other benchmarks. The $\mathsf {SVR}$-trained neural network can handle unseen situations in real-world environments, by leveraging its knowledge learnt from the VR space.
BibTeX
@misc{li2024seamless,
  author = {He Li and Ruihua Han and Zirui Zhao and Wei Xu and Qi Hao and Shuai Wang and Cheng-Zhong Xu},
  title = {{Seamless Virtual Reality With Integrated Synchronizer and Synthesizer for Autonomous Driving}},
  year = {2024},
  eprint = {2403.03541},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.03541},
}
Figure