TacCoRL: Integrating Tactile Feedback into VLA via Simulation

Siyu Ma, Yuqi Liang, Chang Yu, Yunuo Chen, Hao Su, Yixin Zhu, Yin Yang, Chenfanfu JiangarXiv 2026 · arXiv

TL;DR

TacCoRL introduces a scalable framework to inject tactile feedback into pretrained Vision-Language-Action (VLA) models using a real-aligned simulator. By combining sim-real co-training with reinforcement learning anchored by real-world data, it enables robots to learn fine-grained contact corrections for complex bimanual tasks without requiring massive real-world exploration or large-scale tactile pretraining.

What Problem Does This Paper Address?

Pretrained Vision-Language-Action (VLA) models excel at leveraging massive visual and language datasets to guide robot manipulation. However, vision alone falls short in contact-rich manipulation tasks—such as inserting a test tube into a tight rack or assembling interlocking parts. In these scenarios, critical contact regions are frequently occluded by the robot's gripper or the object itself.

While tactile sensing provides the necessary local feedback to resolve these ambiguities, integrating touch into VLAs is notoriously difficult. Large-scale tactile datasets do not exist on the scale of internet vision data, and collecting near-failure recovery data directly on real hardware risks damaging expensive sensors and robots.

Methods

Figure 1: Left: We collect real and simulated visuo-tactile trajectories from aligned real-world and simulation setups. Center: Sim-real co-training gives the policy an initial tactile-conditioned ac- tion prior, and tactile-guided RL in a real-aligned simulator refines closed-loop contact corrections. Right: We deploy the policy directly to the real world, where it achieves high success rates across diverse contact-rich manipulation tasks.

TacCoRL bridges this gap using a two-stage sim-to-real post-training pipeline:

  1. Tactile-Augmented VLA Architecture: Built on top of a $\pi_{0.5}$ VLA backbone, TacCoRL processes a history window of tactile readings through a CNN encoder to produce tactile tokens. A binary contact gate filters out background noise, ensuring tactile tokens only influence the model when actual contact occurs, thereby preserving the VLA's free-space visual priors.
  2. Simulation Alignment (Digital Twin): To train safely, the authors build a high-fidelity simulator aligned with the real world. They use System Identification (SysID) via SPSA optimization to match the real robot's controller response and calibrate a spring-damper tactile model to align simulated contact readings with real ones.
  3. Stage I: Sim-Real Co-training: The model is first fine-tuned via supervised flow-matching on a mixed dataset of real and simulated expert demonstrations. This warm-starts the policy with an initial tactile-conditioned action prior.
  4. Stage II: Tactile-Guided RL with a Real-Data Anchor: The policy is optimized using Proximal Policy Optimization (PPO) in simulation based on sparse task rewards. Crucially, a supervised real-data behavior cloning term acts as an "anchor," preventing the policy from exploiting simulator physics and drifting away from the real-world data distribution.

Datasets

The framework was evaluated on four custom bimanual contact-rich tasks: Test Tube Insertion, Do Puzzle, Assembly #1, and Assembly #2.

The dataset composition per task includes:

  • 50 real-world human demonstrations collected via teleoperation.
  • 20 simulated teleoperation trajectories.
  • 200 synthesized trajectories generated in simulation using MimicGen to bootstrap the reinforcement learning exploration process.

Key Insights and Ideas

  • Don't Just Add Touch—Learn to React: The core value of tactile feedback isn't just knowing contact happened, but knowing how to modulate actions in near-failure states. Simulation provides a safe sandbox to experience these rare, off-nominal states that are missing from standard human demonstrations.
  • The Power of the Real-Data Anchor: Pure simulation RL often leads to sim-to-real gap failures because the policy overfits to simulator quirks. Adding a real-data anchor loss during RL fine-tuning constrains the policy update, keeping it grounded in real-world deployment distributions.
  • Gated Attention Preserves VLA Priors: By ignoring tactile tokens when there is no contact, the VLA's powerful visual-language capabilities remain uncorrupted during free-space movement.

Key Results

  • Real-World Success: Across all four bimanual tasks, TacCoRL achieved an average real-world success rate of 72.5%, significantly outperforming the vision-and-imitation-only baseline average of 50.0%.
  • Task-Specific Gains: In the Assembly #1 task, the success rate jumped from 35% (baseline) to 95% (TacCoRL), demonstrating a massive improvement in handling occluded, tight-tolerance insertions.
  • Ablation Proof: Without the real-data anchor ($\beta=0$), the policy overfit to the simulator, dropping real-world success. A balanced anchor ($\beta=1.0$) yielded the best results, preventing simulator drift while allowing the policy to learn reward-driven contact corrections.

Limitations and Future Work

  • Dependency on Real Tactile Data: While it avoids large-scale pretraining, TacCoRL still requires a small set of real tactile demonstrations to serve as the anchor during post-training.
  • Manual Simulation Setup: Creating the digital twin (aligning cameras, assets, and identifying controller parameters) currently requires significant manual engineering effort.
  • Rigid Contacts Only: The current simulation framework focuses on rigid or near-rigid objects. Tasks involving highly deformable objects or fluids are not yet supported and represent a key direction for future work.

Summary updated

Abstract

Vision-language-action (VLA) models provide strong visual, language, and action priors for robot manipulation, but visual observations alone often miss the local contact state required for contact-rich tasks. We present TacCoRL, a scalable framework that injects Tactile feedback into VLA policies and improves them through sim-real Co-training and simulation-based reinforcement learning (RL), without requiring large-scale tactile pretraining or extensive real-world contact exploration. The key idea is not only adding touch as an input, but learning how contact readings should modulate action responses in near-failure states that are rare in demonstrations and risky to collect on hardware. We use a real-aligned simulator as a closed-loop training environment for contact interaction. Mixed simulated and real trajectories first warm-start tactile-conditioned actions in the pretrained policy. Reinforcement learning with verifiable task rewards then optimizes the policy using simulated contact rollouts. It reinforces tactile-conditioned actions that lead to task completion, while a supervised objective on real trajectories keeps the refined policy anchored to deployment visual, tactile, and action distributions. The resulting policy transfers directly to the real robot without privileged simulation state or online real-world RL. Across four bimanual contact-rich tasks, the final visuo-tactile policy achieves an average success rate of 72.5%, compared to baseline of 50.0%. Result videos and more details are available at https://tac-corl.github.io/
BibTeX
@misc{ma2026taccorl,
  author = {Siyu Ma and Yuqi Liang and Chang Yu and Yunuo Chen and Hao Su and Yixin Zhu and Yin Yang and Chenfanfu Jiang},
  title = {{TacCoRL: Integrating Tactile Feedback into VLA via Simulation}},
  year = {2026},
  eprint = {2606.11743},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.11743},
}

Loading related content...

Figure