Sampling, Communication, and Prediction Co-Design for Synchronizing the Real-World Device and Digital Model in Metaverse

Zhen Meng, Changyang She, Guodong Zhao, Daniele De Martini

TL;DR

This paper introduces a "co-design" framework that jointly optimizes sampling, communication, and prediction to synchronize physical devices with their digital twins in the Metaverse. By using a specialized Deep Reinforcement Learning (DRL) algorithm called KC-TD3, the researchers reduced communication loads by up to 87% while maintaining strict tracking accuracy, even in networks with high packet loss.

What Problem Does This Paper Address?

The "Metaverse" vision relies on seamless, real-time interaction between the physical and virtual worlds. However, achieving this is a massive technical hurdle. To feel "real," synchronization requires Motion-To-Photon (MTP) latency of less than 1ms—a target that even advanced 5G networks struggle to hit consistently.

Current research often tackles parts of the problem in isolation: either optimizing how often we sample data (sampling-communication) or how we guess the next movement (prediction-communication). This paper argues that these three elements—sampling, communication, and prediction—are deeply interdependent. For instance, a better predictor allows for a lower sampling rate, which in turn reduces the strain on the communication network. The authors aim to find the "sweet spot" that minimizes network traffic without sacrificing the accuracy of the digital model.

Methods

The authors propose a Sampling, Communication, and Prediction Co-Design Framework. The heart of this framework is the Knowledge-assisted Constrained Twin-Delayed Deep Deterministic (KC-TD3) policy gradient algorithm.

Key technical innovations include:

  • State-Space Reduction: Instead of feeding raw, high-dimensional trajectory data into the DRL agent (which is slow and complex), they use the Mean Squared Error (MSE) between the real and digital models as a scalar state. This "expert knowledge" significantly speeds up training.
  • Interdependent Action Normalization: The agent decides both the segment length (how long to predict) and the number of samples to send. The algorithm ensures the sampling rate is feasible relative to the segment length.
  • APDO (Accelerated Primal-Dual Optimization): This technique helps the agent learn how to satisfy the "hard" constraint of tracking error while still minimizing the "soft" goal of communication load.
  • Predictor & Reconstructor: They use a Multi-Layer Perceptron (MLP) for trajectory prediction and linear interpolation for reconstructing data from sparse samples.

Datasets

The study did not use a standard public dataset. Instead, the authors built a custom hardware-in-the-loop prototype:

  • Physical Device: A Franka Emika Panda industrial robotic arm with seven degrees of freedom.
  • Digital Model: A virtual replica of the arm built in the Unity game engine.
  • Data Collection: A human operator used a motion capture system (OptiTrack) to guide the arm in drawing a "star" shape. Joint angles were recorded at 1kHz (1,000 samples per second) to create a high-fidelity ground truth dataset for training and evaluation.

Key Insights and Ideas

  • The Power of Synergy: The co-design approach is more flexible than traditional methods. When error constraints are tight, the system acts like a sampling-optimizer; when constraints are loose, it relies more on prediction to save bandwidth.
  • Stability Over "Average" Performance: While a static "exhaustive search" can find a good average sampling rate, the dynamic KC-TD3 policy is much better at handling "tail probabilities"—the rare but jarring spikes in error that ruin a user's immersive experience.
  • Robustness to Loss: The framework is surprisingly resilient. Even with a 10% packet loss rate in the communication link, the co-design approach can still reduce communication load by 75% compared to a baseline system that sends every sample.

Key Results

  • Bandwidth Savings: The framework reduced the average communication load by up to 87% compared to a "Baseline" system (no sampling/prediction) while meeting a tracking error constraint of 0.002°.
  • Accuracy Improvement: Compared to the same baseline, it improved the average tracking error by 87.5% under stringent constraints.
  • Convergence: Thanks to the "knowledge-assisted" features (like state-space reduction), the KC-TD3 algorithm converged to a stable, optimal policy in about 300 episodes.

Limitations and Future Work

  • One-Way Sync: The current prototype focuses on one-way synchronization (physical arm to digital model). Real-world applications like remote surgery or haptic gaming require two-way synchronization (feedback from the digital world to the physical one).
  • Single Device Scaling: The study used one robotic arm. Future work needs to explore how this framework handles multi-user environments where hundreds of devices might be competing for the same network resources.
  • Simplified Network Model: The authors assumed stationary channel gains. In a real mobile environment, the communication channel fluctuates rapidly, which would require the DRL agent to adapt to changing network conditions in real-time.

Summary updated

Abstract

The metaverse has the potential to revolutionize the next generation of the Internet by supporting highly interactive services with satisfactory user experience. The synchronization between devices in the physical world and their digital models in the metaverse is crucial. This work proposes a sampling, communication and prediction co-design framework to minimize the communication load subject to a constraint on the tracking error. To optimize the sampling rate and the prediction horizon, we exploit expert knowledge and develop a constrained deep reinforcement learning algorithm. We validate our framework on a prototype composed of a real-world robotic arm and its digital model. The results show that our framework achieves a better trade-off between the average tracking error and the average communication load compared with a communication system without sampling and prediction. For example, the average communication load can be reduced up to 87% when the average track error constraint is 0.007°. In addition, our policy outperforms the benchmark with the static sampling rate and prediction horizon optimized by exhaustive search, in terms of the tail probability of the tracking error. Furthermore, with the assistance of expert knowledge, the proposed algorithm achieves better convergence time, stability, communication load, and average tacking error.
BibTeX
@article{meng2023sampling,
  author = {Zhen Meng and Changyang She and Guodong Zhao and Daniele De Martini},
  title = {{Sampling, Communication, and Prediction Co-Design for Synchronizing the Real-World Device and Digital Model in Metaverse}},
  journal = {IEEE J. Sel. Areas Commun.},
  year = {2023},
  url = {https://openreview.net/forum?id=ZpnLjDuDWQ},
}

Loading related content...

Figure