TwinBridge: Digital Twin-Driven Policy and Reward Transfer for Real-World Reinforcement Learning in Laboratory Robotics

Shu'ang Sun, Yibo Qiu, Haoliang Ye, Zan Huang, Ao Tian, Kan Ding, Ronald Xu, Mingzhai SunarXiv 2026 · arXiv

TL;DR

TwinBridge is a framework that uses photorealistic digital twins to transfer both robotic policies and dense visual reward models to the real world. By distilling "physics-grounded" supervision from simulation into a visual reward model, robots can master complex laboratory tasks with less than 25 minutes of real-world interaction, achieving a 98% autonomous success rate.

What Problem Does This Paper Address?

Precision laboratory manipulation—like pipetting or handling centrifuge tubes—requires sub-millimeter accuracy and strict adherence to physical prerequisites (e.g., you can't lift a tube until you've secured a grasp). While Reinforcement Learning (RL) is a powerful tool for mastering these skills, it faces two major hurdles in robotics:

  1. Data Scarcity: Collecting enough real-world demonstrations for multi-stage tasks is labor-intensive.
  2. Sparse Rewards: In the real world, a robot often only knows if it succeeded or failed at the very end of an episode. Without "dense" feedback on intermediate progress, the learner struggles to understand which specific actions led to success.

While "Sim-to-Real" transfer helps, most methods focus only on transferring the policy. TwinBridge argues that we should also transfer the supervision—the ability to recognize and reward progress—from the digital twin to the physical robot.

Methods

The TwinBridge framework consists of three main phases:

  1. Digital Twin Construction: The authors combine 3D Gaussian Splatting (3DGS) for photorealistic rendering with a physics engine (Isaac Sim). This creates a high-fidelity environment where the robot can "see" and "feel" interactions that closely match the real world.
  2. Physically Grounded Reward Distillation: In simulation, the system has access to ground-truth states (contacts, alignment, depths). They use an execution-history ledger to define rewards based on ordered events (e.g., Event A must happen before Event B) and geometric progress. This supervision is distilled into a visual reward model (a causal GRU with a frozen Octo encoder) trained on simulated images and a small handful of annotated real-world images.
  3. Real-World Online RL: The robot starts with a policy pre-trained in simulation. During real-world interaction, the frozen visual reward model provides dense feedback. The system uses a human-in-the-loop (HIL) approach where an operator can intervene to correct the robot, and these corrections are used to refine the policy online.
Overview of the TwinBridge framework, showing the pipeline from digital twin pre-training to real-world refinement.

Datasets

The authors evaluated TwinBridge on five representative laboratory tasks:

  • Centrifuge tube pickup: Lifting a tube from a rack.
  • Petri dish lid removal: Separating and relocating a lid.
  • Centrifuge tube loading: Inserting a tube into a centrifuge holder.
  • Pipette tip attachment: Picking up a disposable tip.
  • Pipetting: A complex sequence involving aspiration, transfer, and dispensing.

Evaluation was performed on a JAKA Minicobo robot arm. The reward model was trained using 40 simulation episodes per task and a "small set" of real visual annotations.

Key Insights and Ideas

  • Transferring the "Coach," not just the "Player": The core innovation is using the digital twin to train a reward model that can then "coach" the robot in the real world. This bypasses the need for complex real-world instrumentation to measure progress.
  • The Ledger Prevents "Reward Hacking": A common problem in RL is the robot finding ways to get rewards without actually doing the task (e.g., jittering near a goal). The execution-history ledger ensures the robot only gets credit for new valid progress and credits each milestone only once per episode.
  • 3DGS Bridges the Visual Gap: By using 3D Gaussian Splatting, the simulated images are realistic enough that a reward model trained on them can generalize to real camera feeds with very little real-world data.
The reward construction process, showing how simulator states are turned into a visual reward model.

Key Results

  • High Success, Low Time: TwinBridge achieved a 98% mean autonomous success rate across all five tasks. Crucially, this was achieved with $\le$ 25 minutes of real-world interaction time per task.
  • Dense vs. Sparse: In simulation, the dense rewards provided by TwinBridge allowed the robot to reach 90-100% success in tasks where sparse-reward baselines peaked at only 50% (see Figure 4).
  • Superior to Baselines: Compared to HIL-ConRFT (which uses sparse rewards), TwinBridge improved success rates by 30 to 70 percentage points across the different tasks. It also significantly outperformed imitation-learning-based methods like HG-DAgger.
  • Robust Visual Tracking: The visual reward model achieved a 99.3% F1 score for event detection on real-world trajectories, proving it can accurately track task milestones despite the sim-to-real gap.
Success rates in simulation, comparing TwinBridge's dense rewards to sparse rewards.

Limitations and Future Work

  • Setup Overhead: Creating a high-fidelity digital twin (3DGS scanning plus manual physics modeling of lab equipment) requires significant upfront effort.
  • Human-in-the-Loop Dependency: While the final policy is autonomous, the training phase still requires a human operator to be present for interventions.
  • Future Directions: The authors suggest scaling this to longer-horizon "autonomous scientist" workflows and exploring ways to further reduce the need for real-world annotations during reward distillation.

Summary updated

Abstract

Online reinforcement learning offers an effective approach to precise laboratory manipulation, yet costly real-world demonstrations and limited access to dense process rewards hinder its practical application. We present TwinBridge, a digital twin-driven framework that supplies policy initialization and dense visual rewards for real-world online learning. We construct digital twins by combining 3D Gaussian splatting with physics simulation. Here, simulator states provide task events and geometric progress under physical prerequisites and ordering constraints, yielding rewards for policy pretraining with simulated demonstrations and reinforcement learning. We distill this supervision into a visual reward model. During real-world interaction, its predictions are combined with execution history to compute dense rewards for human-in-the-loop policy refinement. Across five real-world laboratory tasks, TwinBridge achieves a final mean autonomous success rate of 98% with at most 25 minutes of real-world interaction per task. Experiments show that dense rewards improve interaction efficiency over sparse rewards, and simulation-based policy initialization further accelerates learning.
BibTeX
@misc{sun2026twinbridge,
  author = {Shu'ang Sun and Yibo Qiu and Haoliang Ye and Zan Huang and Ao Tian and Kan Ding and Ronald Xu and Mingzhai Sun},
  title = {{TwinBridge: Digital Twin-Driven Policy and Reward Transfer for Real-World Reinforcement Learning in Laboratory Robotics}},
  year = {2026},
  eprint = {2611.06442},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06442},
}
Figure