Sampling, Communication, and Prediction Co-Design for Synchronizing the Real-World Device and Digital Model in Metaverse

Zhen Meng, Changyang She, Guodong Zhao, Daniele De Martini

Abstract

The metaverse has the potential to revolutionize the next generation of the Internet by supporting highly interactive services with the help of Mixed Reality (MR) technologies; still, to provide a satisfactory experience for users, the synchronization between the physical world and its digital models is crucial. This work proposes a sampling, communication and prediction co-design framework to minimize the communication load subject to a constraint on tracking the Mean Squared Error (MSE) between a real-world device and its digital model in the metaverse. To optimize the sampling rate and the prediction horizon, we exploit expert knowledge and develop a constrained Deep Reinforcement Learning (DRL) algorithm, named Knowledge-assisted Constrained Twin-Delayed Deep Deterministic (KC-TD3) policy gradient algorithm. We validate our framework on a prototype composed of a real-world robotic arm and its digital model.

Introduction

The "Metaverse" vision relies on seamless, real-time interaction between the physical and virtual worlds. However, achieving this is a massive technical hurdle. To feel "real," synchronization requires Motion-To-Photon (MTP) latency of less than 1ms—a target that even advanced 5G networks struggle to hit consistently.

Current research often tackles parts of the problem in isolation: either optimizing how often we sample data (sampling-communication) or how we guess the next movement (prediction-communication). This paper argues that these three elements—sampling, communication, and prediction—are deeply interdependent. For instance, a better predictor allows for a lower sampling rate, which in turn reduces the strain on the communication network. The authors aim to find the "sweet spot" that minimizes network traffic without sacrificing the accuracy of the digital model.

The Co-Design Framework

The authors propose a Sampling, Communication, and Prediction Co-Design Framework to optimize the synchronization process.

The proposed co-design framework synchronizing the Real World and Metaverse.

As illustrated in the framework, the physical world trajectory is sensed and sampled. These samples are transmitted via a forward link to the Metaverse, where the digital model is reconstructed and future states are predicted. A feedback link allows for dynamic adjustment of parameters based on the current state and tracking error, ensuring the virtual model remains a high-fidelity twin of the physical device.

Knowledge-assisted Constrained DRL (KC-TD3)

The heart of this framework is the Knowledge-assisted Constrained Twin-Delayed Deep Deterministic (KC-TD3) policy gradient algorithm.

Architecture of the KC-TD3 algorithm and its interaction with the device and virtual model.

Key technical innovations include:

  • State-Space Reduction: Instead of feeding raw, high-dimensional trajectory data into the DRL agent, they use the Mean Squared Error (MSE) between the real and digital models as a scalar state. This "expert knowledge" significantly speeds up training and improves convergence.
  • Interdependent Action Normalization: The agent decides both the segment length (how long to predict) and the number of samples to send. The algorithm ensures the sampling rate is feasible relative to the segment length.
  • APDO (Accelerated Primal-Dual Optimization): This technique helps the agent learn how to satisfy the "hard" constraint of tracking error while still minimizing the "soft" goal of communication load.

Experimental Validation and Results

The study utilized a custom hardware-in-the-loop prototype featuring a Franka Emika Panda industrial robotic arm and a virtual replica built in the Unity engine. A human operator guided the arm using a motion capture system to create a high-fidelity ground truth dataset for evaluation.

Key results include:

  • Bandwidth Savings: The framework reduced the average communication load by up to 87% compared to a baseline system without sampling/prediction while meeting a tracking error constraint of 0.002°.
  • Adaptability: The framework is highly flexible. Under stringent error constraints (MSE=0.002°), the policy prioritizes sampling; under milder constraints (MSE=0.007°), it relies more on prediction to save bandwidth.
  • Robustness to Loss: Even with a 10% packet loss rate in the communication link, the co-design approach reduced communication load by 75% compared to sending every sample.
  • Tail Probability: The dynamic KC-TD3 policy is significantly better at handling "tail probabilities"—rare but jarring spikes in error—compared to static optimization methods found via exhaustive search.

Conclusion

This co-design approach demonstrates that joint optimization of sampling, communication, and prediction is essential for efficient Metaverse synchronization. By leveraging expert knowledge within a DRL framework, the system achieves high accuracy with minimal network overhead, providing a robust foundation for future interactive Mixed Reality services.

Summary updated

Abstract

The metaverse has the potential to revolutionize the next generation of the Internet by supporting highly interactive services with the help of Mixed Reality (MR) technologies; still, to provide a satisfactory experience for users, the synchronization between the physical world and its digital models is crucial. This work proposes a sampling, communication and prediction co-design framework to minimize the communication load subject to a constraint on tracking the Mean Squared Error (MSE) between a real-world device and its digital model in the metaverse. To optimize the sampling rate and the prediction horizon, we exploit expert knowledge and develop a constrained Deep Reinforcement Learning (DRL) algorithm, named Knowledge-assisted Constrained Twin-Delayed Deep Deterministic (KC-TD3) policy gradient algorithm. We validate our framework on a prototype composed of a real-world robotic arm and its digital model. Compared with existing approaches: (1) When the tracking error constraint is stringent (MSE=0.002 degrees), our policy degenerates into the policy in the sampling-communication co-design framework. (2) When the tracking error constraint is mild (MSE=0.007 degrees), our policy degenerates into the policy in the prediction-communication co-design framework. (3) Our framework achieves a better trade-off between the average MSE and the average communication load compared with a communication system without sampling and prediction. For example, the average communication load can be reduced up to 87% when the track error constraint is 0.002 degrees. (4) Our policy outperforms the benchmark with the static sampling rate and prediction horizon optimized by exhaustive search, in terms of the tail probability of the tracking error. Furthermore, with the assistance of expert knowledge, the proposed algorithm KC-TD3 achieves better convergence time, stability, and final policy performance.
BibTeX
@article{meng2023sampling,
  author = {Zhen Meng and Changyang She and Guodong Zhao and Daniele De Martini},
  title = {{Sampling, Communication, and Prediction Co-Design for Synchronizing the Real-World Device and Digital Model in Metaverse}},
  journal = {IEEE J. Sel. Areas Commun.},
  year = {2023},
  url = {https://openreview.net/forum?id=ZpnLjDuDWQ},
}
Figure