Generative AI-empowered Effective Physical-Virtual Synchronization in the Vehicular Metaverse

Minrui Xu, Dusit Niyato, Hongliang Zhang, Jiawen Kang, Zehui Xiong, Shiwen Mao, Zhu Han

Abstract

Metaverse seamlessly blends the physical world and virtual space via ubiquitous communication and computing infrastructure. In transportation systems, the vehicular Metaverse can provide a fully-immersive and hyperreal traveling experience (e.g., via augmented reality head-up displays, AR-HUDs) to drivers and users in autonomous vehicles (AVs) via roadside units (RSUs). However, provisioning real-time and immersive services necessitates effective physical-virtual synchronization between physical and virtual entities, i.e., AVs and Metaverse AR recommenders (MARs). In this paper, we propose a generative AI-empowered physical-virtual synchronization framework for the vehicular Metaverse. In physical-to-virtual synchronization, digital twin (DT) tasks generated by AVs are offloaded for execution in RSU with future route generation. In virtual-to-physical synchronization, MARs customize diverse and personal AR recommendations via generative AI models based on user preferences. Furthermore, we propose a multi-task enhanced auction-based mechanism to match and price AVs and MARs for RSUs to provision real-time and effective services. Finally, property analysis and experimental results demonstrate that the proposed mechanism is strategy-proof and adverse-selection free while increasing social surplus by 50%.

Introduction

The "Vehicular Metaverse" represents a paradigm shift in transportation, where autonomous vehicles (AVs) serve as portals to an immersive digital layer. This integration requires a bidirectional synchronization process: Physical-to-Virtual (P2V), where vehicle data updates a "Digital Twin" (DT) for predictive modeling, and Virtual-to-Physical (V2P), where the Metaverse delivers personalized Augmented Reality (AR) content to the vehicle's head-up displays (HUDs).

However, realizing this vision faces significant hurdles. AVs often lack the onboard computational power for real-time high-fidelity rendering, and creating personalized AR content for every user manually is unfeasible. This research addresses these challenges by leveraging Generative AI and a novel economic mechanism to facilitate efficient resource allocation and content creation.

Overview of the physical-virtual synchronization framework in the vehicular Metaverse.

Proposed Framework: Physical-Virtual Synchronization

The proposed framework treats synchronization as a dual-task offloading problem managed by Roadside Units (RSUs):

  1. P2V Synchronization: AVs offload DT tasks to RSUs. These units process sensor data to maintain accurate digital representations and predict future trajectories.
  2. V2P Synchronization: Metaverse AR Recommenders (MARs) utilize these digital twins to understand user preferences and context. They then generate customized AR recommendations—ranging from functional navigation aids to infotainment—which are delivered back to the AV.

Generative AI for Content Customization

To solve the content scarcity problem, the authors introduce TSDreamBooth, a specialized generative AI model. Adapted from DreamBooth (a fine-tuned Stable Diffusion model), TSDreamBooth is trained on traffic sign datasets (such as the BelgiumTS dataset) to generate high-quality, personalized AI-Generated Content (AIGC).

Qualitative results of TSDreamBooth showing background modification and re-contextualization of traffic signs for AR displays.

TSDreamBooth allows the system to modify traffic signs on the fly—changing background colors or placing them in different environmental contexts like snow or coastal settings—while maintaining high recognition accuracy. This ensures that AR overlays are not only aesthetically pleasing and personalized but also functionally reliable for the driver.

Economic Mechanism: MTEPViSA Auction

Because RSUs and MARs are independent entities providing services, an economic framework is needed to incentivize participation and ensure fair pricing. The researchers developed the Multi-Task Enhanced Physical-Virtual Synchronization Auction (MTEPViSA).

This second-score auction mechanism is designed with two critical properties:

  • Strategy-proofness: It ensures that participants (AVs and MARs) maximize their utility by bidding their true valuations, preventing market manipulation.
  • Adverse-selection freedom: It filters out low-quality service providers, ensuring that the synchronization remains high-quality and reliable.

Experimental Results and Insights

The framework was evaluated using the Belgium Traffic Sign dataset and simulated vehicular environments. Key findings include:

  • Increased Social Surplus: The MTEPViSA mechanism achieved a 50% increase in social surplus compared to standard auction models, indicating a more efficient allocation of resources.
  • High Visual Fidelity: TSDreamBooth maintained recognition accuracy as high as 1.00 for several sign types, proving that generative AI can produce assets that are both diverse and functional.
  • Real-time Efficiency: The multi-task offloading approach successfully managed strict deadlines for DT updates, ensuring seamless synchronization even under computational constraints.

Conclusion

By combining Generative AI with a robust auction-based resource allocation mechanism, this framework provides a foundation for a scalable and personalized vehicular Metaverse. Future work will focus on optimizing these diffusion models for edge computing environments and improving performance in areas with sparse infrastructure.

Summary updated

Abstract

Metaverse seamlessly blends the physical world and virtual space via ubiquitous communication and computing infrastructure. In transportation systems, the vehicular Metaverse can provide a fully-immersive and hyperreal traveling experience (e.g., via augmented reality head-up displays, AR-HUDs) to drivers and users in autonomous vehicles (AVs) via roadside units (RSUs). However, provisioning real-time and immersive services necessitates effective physical-virtual synchronization between physical and virtual entities, i.e., AVs and Metaverse AR recommenders (MARs). In this paper, we propose a generative AI-empowered physical-virtual synchronization framework for the vehicular Metaverse. In physical-to-virtual synchronization, digital twin (DT) tasks generated by AVs are offloaded for execution in RSU with future route generation. In virtual-to-physical synchronization, MARs customize diverse and personal AR recommendations via generative AI models based on user preferences. Furthermore, we propose a multi-task enhanced auction-based mechanism to match and price AVs and MARs for RSUs to provision real-time and effective services. Finally, property analysis and experimental results demonstrate that the proposed mechanism is strategy-proof and adverse-selection free while increasing social surplus by 50%.
BibTeX
@inproceedings{xu2023generative,
  author = {Minrui Xu and Dusit Niyato and Hongliang Zhang and Jiawen Kang and Zehui Xiong and Shiwen Mao and Zhu Han},
  title = {{Generative AI-empowered Effective Physical-Virtual Synchronization in the Vehicular Metaverse}},
  booktitle = {Metacom},
  year = {2023},
  doi = {10.1109/metacom57706.2023.00106},
}
Figure