TouchTwin: Scaling Human Touch Data via Real-to-Sim-to-Real

Huiwen Han, Nuo Chen, Ziyao Zeng, Dayou Li, Hao Wang, Amir Hossein Rahmati, Xiaoning Qian, Bangya Liu, Zhiwen FanarXiv 2026 · arXiv

TL;DR

TouchTwin is a "real-to-sim-to-real" pipeline that uses tactile glove data to bridge the "force gap" in dexterous manipulation. By grounding physics simulations in actual human touch measurements, it generates massive amounts of physically accurate, augmented demonstration data—complete with realistic video—that significantly outperforms vision-only methods in training robust robotic policies.

What Problem Does This Paper Address?

In the quest to teach robots dexterous manipulation, human demonstrations are gold. However, collecting enough high-quality data is a bottleneck. Existing "real-to-sim" pipelines try to solve this by rebuilding human videos in simulation to create infinite variations.

The catch? These pipelines rely almost entirely on vision. While cameras are great at showing where a hand moves, they are blind to how hard it is pressing. This creates a "force gap." A hand pose in a video only shows where a finger stops on a surface; it doesn't show the internal force driving that contact. Consequently, policies trained on vision-only data tend to either "ghost" objects (gripping with only 31% of the required force) or over-grip them (155% of the force), leading to fragile performance in the real world.

Methods

TouchTwin introduces an agentic pipeline to rebuild human interactions in the MuJoCo physics engine with high fidelity.

  1. Agentic Real2Sim: The system uses LLM-based coding agents to write physics-tested programs for objects, ensuring they have the correct joints, latches, and properties (see Figure 3).
  2. Tactile Hand Modeling: For every human subject, the researchers build a personalized hand model. This model includes a simulated version of the "TouchScale" glove, featuring 880 "taxels" (tactile pixels) distributed across 15 pads (see Figure 2).
  3. Touch-Grounded Replay: Instead of just following the visual trajectory, the simulation is driven by the recorded tactile data. This data tells the simulator exactly where contact occurs, when it starts, and how hard the fingers are pressing.
  4. Data Scaling via Perturbations: Once a recording is successfully "twinned" in simulation, the system generates $N$ perturbations—changing object mass, friction, or hand posture.
  5. Video Generation: To turn these simulated replays back into training data for vision-based robots, TouchTwin uses a video diffusion model (Wan2.2). The model is conditioned on the simulation to generate realistic video that obeys the laws of physics.
The TouchTwin overview showing the agentic pipeline and data generation process.

Datasets

The primary source of data is TouchScale, a recently introduced dataset containing 500 hours of paired egocentric vision (head and wrist RGB-D) and tactile glove data. TouchTwin uses these recordings to seed its simulation environment.

Key Insights and Ideas

  • The Force Gap is Quantifiable: The authors show that vision-only reconstruction is fundamentally flawed for force estimation. Without tactile grounding, you simply cannot distinguish a light touch from a firm grip.
  • Touch as a "Physics Checker": In the agentic pipeline, the tactile data acts as a ground-truth validator. If the simulation's contact forces don't match the glove's recorded forces, the "review agent" triggers a repair of the simulation parameters.
  • Simulation Fills the Gaps: Interestingly, the simulation can estimate forces in areas where the physical glove has no sensors, such as the center of the palm. By "twining" the glove's data, the simulator provides a complete force map of the entire hand (see Figure 9).
  • Physics-Conditioned Video: Standard video generation often suffers from "hallucinations" where objects float or clip through hands. By conditioning the video model on a physics-accurate simulation, TouchTwin ensures the generated training videos are visually realistic and physically consistent.
Comparison of real glove force vs. simulated contact force, showing near-perfect alignment.

Key Results

The inclusion of touch data leads to dramatic improvements across the board:

  • Grip Accuracy: Touch-grounded replays achieved a grip force of 0.99x the recorded human force, compared to 0.31x for pose imitation and 1.55x for vision-based physics replays.
  • Policy Success: A hand policy trained on TouchTwin data succeeded in 20/20 trials on held-out recordings and 35/36 unseen perturbations.
  • Contact Prediction: A model trained on these replays missed only 13% of real contact frames, a significant improvement over the 32% miss rate of vision-only baselines.
  • Visual Fidelity: In video generation tasks, conditioning on the simulation improved the SSIM (Structural Similarity Index) from 0.66 to 0.76 compared to text-only baselines.

Limitations and Future Work

While a major step forward, the authors note several limitations:

  • Task Diversity: Most experiments focused on a mug pick-up task; broader task diversification is needed.
  • Sim-to-Real Gap: All policies were tested within the simulation environment. True "real-to-sim-to-real" transfer to physical robot hands remains a critical next step.
  • Small Scale: The held-out evaluation sets are currently small, and the system still struggles with complex two-handed slips (12–23 mm of drift).
  • Open-Loop Testing: The policies currently act in an open-loop fashion; future iterations will likely need closed-loop tactile feedback to handle dynamic environments.

Summary updated

Abstract

Human demonstrations are becoming the main source of data for dexterous manipulation, and real-to-sim-to-real pipelines multiply each demonstration in simulation. These pipelines rebuild the interaction from vision, which recovers how the hand moves. Which fingers touch, when, and how hard remain unobserved, because a hand pose shows where each finger stops on the surface rather than how far it is driven to close. This gap is a force gap: on held-out recordings, a hand policy that imitates the observed pose grips with 0.31x the recorded force, and a policy trained on physics replays of vision-tracked poses grips with 1.55x. TouchTwin closes the gap with measured human touch. Each selected TouchScale recording, captured with head and wrist RGB-D cameras and an 880-taxel glove, is rebuilt in MuJoCo. Coding agents write the objects as physics-tested programs, and a per-subject tactile hand replays the interaction with the recorded touch setting the contact and the closing force of each finger. In simulation, a policy trained on touch-grounded replays grips at 0.99x the recorded force and succeeds in 20/20 trials on held-out recordings and 35/36 unseen perturbations; a contact predictor trained on them misses 13% of real contact frames, against 32% for vision-based replays. Each replay also seeds perturbed demonstrations with simulated contact and force labels, rendered as realistic video in new views and scenes.
BibTeX
@misc{han2026touchtwin,
  author = {Huiwen Han and Nuo Chen and Ziyao Zeng and Dayou Li and Hao Wang and Amir Hossein Rahmati and Xiaoning Qian and Bangya Liu and Zhiwen Fan},
  title = {{TouchTwin: Scaling Human Touch Data via Real-to-Sim-to-Real}},
  year = {2026},
  eprint = {2611.17619},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17619},
}
Figure