Demonstrating DVS: Dynamic Virtual-Real Simulation Platform for Mobile Robotic Tasks

Zijie Zheng, Zhixin Li, Yunpeng Wang, Qiang Xie, Long Zeng2025 · DOI

TL;DR

DVS (Dynamic Virtual-Real Simulation) is a new platform designed to bridge the "sim-to-real" gap by providing bidirectional, real-time synchronization between virtual environments and physical robots. By integrating stochastic pedestrian modeling and a high-precision motion capture workflow, it enables researchers to develop and test mobile robotic tasks—like social navigation and complex manipulation—in dynamic, unpredictable indoor settings.

What Problem Does This Paper Address?

While robotic simulation has advanced significantly, most existing platforms (like Habitat or iGibson) focus on static environments or scripted interactions. This creates a major hurdle for "embodied AI" because real-world environments are messy, dynamic, and unpredictable.

The authors identify three specific gaps:

  1. The Sim-to-Real Gap: Algorithms that perform perfectly in a simulator often fail in the real world due to sensor noise and physical latencies.
  2. Static Scenes: Most simulators lack realistic, socially-aware pedestrian movement, which is critical for indoor mobile robots.
  3. Lack of Interoperability: There is rarely a seamless way to synchronize a physical robot’s state with its virtual twin in real-time, making it difficult to perform "human-in-the-loop" interventions or closed-loop validation.

Methods

The DVS platform introduces a "Virtual-Real Fusion" architecture built on three pillars:

  • Bidirectional Synchronization: Using an optical motion capture system (VRPN) and ROS (Robot Operating System), the platform achieves sub-millimeter precision (0.1 mm) and high angular accuracy (0.1°) when aligning physical robots with their virtual counterparts.
  • Dynamic Scene Generation: A dedicated plug-in populates virtual scenes with "mobile robot proxies" and pedestrians. These agents aren't just moving in straight lines; they feature variable accelerations and socially compliant collision avoidance.
  • Intervention-Enabled Workflow: Unlike "set-and-forget" simulations, DVS allows researchers to interrupt a robot mid-task. For example, if a robot is picking up an apple, a researcher can change the target to a banana in the virtual scene, and the physical robot will adapt its trajectory based on the updated virtual state.
  • Data Pipeline Optimization: To improve the quality of synthetic training data, the authors use Bézier curves to smooth camera trajectories, reducing abrupt directional changes that typically degrade 3D reconstruction and feature matching.

Datasets

The paper utilizes several environments and benchmarks:

  • Synthetic Indoor Scenes: Custom-built high-fidelity environments including a Gym, Office, Supermarket, Restaurant, and Villa.
  • Public Benchmarks: The platform's pedestrian models were compared against the ETH outdoor dataset to highlight the increased complexity of indoor navigation.
  • Task-Specific Data: The authors generated a large-scale virtual-real fusion dataset for grasping, navigation, and trajectory prediction, featuring RGB, depth, 2D/3D bounding boxes, and semantic segmentation.

Key Insights and Ideas

  • Indoor vs. Outdoor Complexity: Trajectory prediction algorithms (like STGAT) that work well outdoors see their error rates (ADE/FDE) nearly double when moved to crowded indoor environments. This is due to dense static obstacles and more frequent, unpredictable human interactions.
  • Smoothing is Crucial for Vision: Simply smoothing the camera path using Bézier curves nearly doubled the number of detectable features in complex scenes (e.g., from 751 to 1484 features in a bedroom scene), proving that "robotic" jerky movement is a major bottleneck for synthetic data quality.
  • The Power of Intervention: The study shows that large vision-language models (like OpenVLA) often fail in heterogeneous deployments. However, by using the DVS "intervention" mechanism, researchers can correct a robot's behavior mid-execution, achieving success rates that would be impossible with a purely autonomous, pre-trained policy.

Key Results

  • Trajectory Prediction: In indoor tests, the TUTR (Transformer-based) model outperformed graph-based methods (STGAT), showing better resilience to the complex spatial-temporal interactions of indoor crowds.
  • Social Navigation: Testing algorithms like AttnGraph in the DVS Restaurant scene showed that as the number of pedestrians increased from 10 to 20, the success rate dropped from 83% to 67%, highlighting the platform's ability to stress-test navigation stacks.
  • Sim-to-Real Grasping: Robots fine-tuned with "virtual-real" synchronized data achieved higher success rates (up to 90%) compared to those trained on purely virtual data (60%), demonstrating that the feedback loop effectively bridges the reality gap.

Limitations and Future Work

The authors acknowledge several areas for growth:

  • Haptic Feedback: The current system focuses on visual and positional synchronization but lacks the haptic/force feedback necessary for more delicate manipulation tasks.
  • Automation of Intervention: While the platform supports human intervention, the authors aim to develop AI-driven intervention strategies to automate the error-correction process.
  • Hardware Compatibility: Future versions will focus on expanding compatibility beyond the Kinova Gen3 arm to a wider variety of industrial and collaborative robotic platforms.

Summary updated

BibTeX
@misc{zheng2025demonstrating,
  author = {Zijie Zheng and Zhixin Li and Yunpeng Wang and Qiang Xie and Long Zeng},
  title = {{Demonstrating DVS: Dynamic Virtual-Real Simulation Platform for Mobile Robotic Tasks}},
  year = {2025},
  doi = {10.15607/rss.2025.xxi.129},
}
Figure