Abstract
Real-time Digital Twinning of physical world scenes onto the Metaverse is necessary for a myriad of applications such as augmented-reality (AR) assisted driving. In AR assisted driving, physical environment scenes are first captured by Internet of Vehicles (IoVs) and are uploaded to the Metaverse. A central Metaverse Map Service Provider (MMSP) will aggregate information from all IoVs to develop a central Metaverse Map. Information from the Metaverse Map can then be downloaded into individual IoVs on demand and be delivered as AR scenes to the driver. However, the growing interest in developing AR assisted driving applications which relies on digital twinning invites adversaries. These adversaries may place physical adversarial patches on physical world objects such as cars, signboards, or on roads, seeking to contort the virtual world digital twin. Hence, there is a need to detect these physical world adversarial patches. Nevertheless, as real-time, accurate detection of adversarial patches is compute-intensive, these physical world scenes have to be offloaded to the Metaverse Map Base Stations (MMBS) for computation. Hence in our work, we considered an environment with moving Internet of Vehicles (IoV), uploading real-time physical world scenes to the MMBSs. We formulated a realistic joint variable optimization problem where the MMSPs' objective is to maximize adversarial patch detection mean average precision (mAP), while minimizing the computed AR scene up-link transmission latency and IoVs' up-link transmission idle count, through optimizing the IoV-MMBS allocation and IoV up-link scene resolution selection. We proposed a Heterogeneous Action Proximal Policy Optimization (HAPPO) (discrete-continuous) algorithm to tackle the proposed problem. Extensive experiments shows HAPPO outperforms baseline models when compared against key metrics.
Introduction
As the Metaverse evolves from a conceptual framework into a functional ecosystem, applications like Augmented Reality (AR)-assisted driving are becoming increasingly prominent. These systems rely on "Digital Twinning"—the real-time synchronization of physical world scenes into a virtual map. In this architecture, Internet of Vehicles (IoV) sensors capture environmental data and upload it to Metaverse Map Service Providers (MMSP). This data is then aggregated to create a comprehensive virtual map, which is subsequently delivered back to vehicles as AR overlays to assist drivers.
However, this pipeline introduces significant security vulnerabilities. Adversaries can deploy physical adversarial patches—inconspicuous stickers placed on road signs or vehicles—designed to deceive AI-based detection systems. These patches can corrupt the digital twin, leading to misinformation or dangerous driving decisions. Detecting these threats is computationally expensive and requires high-resolution imagery, creating a critical bottleneck at the network edge.
The Edge Computing Dilemma
The challenge of securing the Metaverse digital twin lies in a fundamental trade-off between security and network performance:
- High Resolution: Essential for high detection accuracy (mean Average Precision, or mAP), but leads to increased up-link transmission latency and network interference.
- Low Resolution: Reduces latency and bandwidth consumption but renders adversarial patches nearly invisible to AI detectors.
To address this, the researchers formulated a joint optimization problem aimed at maximizing detection mAP while minimizing transmission latency and the "idle count" (the frequency with which vehicles are unable to upload data due to network constraints). This requires dynamically optimizing both the IoV-to-Base-Station allocation and the specific image resolution used for each upload.
Methodology: The HAPPO Algorithm
Standard Deep Reinforcement Learning (DRL) algorithms often struggle with "heterogeneous" action spaces, where a system must simultaneously make categorical decisions (e.g., selecting a base station) and numerical decisions (e.g., selecting a resolution between 64 and 416 ppi). To solve this, the authors proposed the Heterogeneous Action Proximal Policy Optimization (HAPPO) algorithm.
Dual-Agent Architecture
HAPPO utilizes two specialized agents working in tandem:
- Agent 1 (Discrete): Manages the allocation of IoVs to Metaverse Map Base Stations (MMBS).
- Agent 2 (Continuous): Determines the optimal physical scene resolution for transmission.
Detection and Modeling
The system employs a YOLOv4 object detector with a CSPDarknet53 backbone to identify adversarial patches. Because running a full detector within an RL training loop is computationally prohibitive, the researchers established a polynomial relationship between image resolution and detection mAP. This empirical model allows the RL agents to simulate the security impact of their decisions with high efficiency.
Framework
The system adopts a Centralized Training and Decentralized Execution (CTDE) architecture. This ensures the model benefits from global network knowledge during the training phase while remaining capable of making rapid, localized decisions at the edge during deployment.
Experimental Setup and Datasets
The study utilized two primary datasets to train and evaluate the adversarial patch detection model:
- Stanford-Cars: 2,000 images of various vehicle types.
- nuImages: 2,000 images from a large-scale autonomous driving dataset.
To simulate real-world threats, adversarial patches were digitally inserted onto these images using the Projected Gradient Descent (PGD) attack method. This created a robust testing ground for the HAPPO algorithm's ability to maintain security under varying network conditions.
Key Results and Insights
Superior Performance and Scalability
HAPPO consistently outperformed baseline models, including Independent PPO and Heterogeneous A2C (HAA2C). In high-congestion scenarios (e.g., seven vehicles sharing three base stations), HAPPO maintained significantly higher rewards, demonstrating its superior ability to manage network interference.
Stability and Convergence
A critical finding was the stability of the HAPPO approach. The algorithm exhibited a much narrower range of variance in transmission time and mAP compared to baselines, suggesting a more predictable and reliable service for real-time driving. Furthermore, HAPPO reached optimal performance levels significantly faster during the training phase than independent agent approaches.
The Importance of "Idle Count"
Beyond speed and accuracy, the authors highlighted the "Idle Count" metric. If a system excludes too many vehicles to preserve bandwidth, the resulting digital twin becomes incomplete and potentially hazardous. HAPPO’s ability to minimize these exclusions ensures a more comprehensive and safer virtual representation of the physical world.
Conclusion and Future Work
This research provides a robust framework for securing the Metaverse's digital twins against physical-world adversarial attacks. By leveraging heterogeneous DRL, the HAPPO algorithm successfully balances the competing demands of high-resolution security detection and low-latency mobile networking.
Future research directions include incorporating more realistic urban topology and traffic flow models, exploring dynamic edge infrastructure (such as drone-based base stations), and validating the system against actual physical stickers in a real-world testbed.
Summary updated