TacInno: Teaching Vision-Language-Action Models the Contact Dynamics Beyond Vision and Action

Weile Li, Yunfan Lou, Muqing Shi, Zihao Zhang, Zhenbiao Huang, Boxian Yao, Xiyu Zhu, Zhibo PangarXiv 2026 · arXiv

In the quest to build general-purpose robots, Vision-Language-Action (VLA) models have become the gold standard. However, even the most advanced VLAs struggle with "contact-rich" tasks—things like inserting a tiny plug or unscrewing a tight bottle cap—where visual information is often occluded or too subtle to be useful. While adding tactile sensors seems like an obvious fix, it hasn't always worked as expected.

A new paper, "TacInno: Teaching Vision-Language-Action Models the Contact Dynamics Beyond Vision and Action," identifies a surprising reason for this: most tactile data is actually redundant. By isolating the "innovation"—the part of the tactile signal that vision and actions can't predict—the authors have developed a method that significantly boosts robot performance in complex manipulation.

TL;DR

TacInno introduces "tactile innovation"—the residual tactile information that cannot be predicted from visual observations or robot actions—as a targeted supervision signal. By training VLA models to predict this "innovation" as an auxiliary task, the researchers achieved a 10.7% success rate boost in simulation and 9.0% in the real world, while ensuring the models truly rely on their tactile sense.

What Problem Does This Paper Address?

Equipping robots with vision-based tactile sensors (like GelSight) provides a wealth of data about contact forces and slips. Yet, many existing visuo-tactile models show only modest gains over vision-only baselines.

The authors argue that this is because a huge portion of the tactile signal is redundant. For example, when a robot closes its gripper, the resulting gel deformation is almost entirely predictable from the "close" command and the visual sight of the gripper moving. If a model is trained to predict the entire future tactile state, it spends most of its "brainpower" learning these obvious, action-induced changes, often ignoring the subtle, critical signals like the start of a slip.

Overview of TacInno: isolating the contact response that only tactile input reveals.

Methods

The core contribution is the TacInno Tokenizer, a conditional VQ-VAE (Vector Quantized Variational Autoencoder) designed to extract "tactile innovation."

  1. Isolating Innovation: The tokenizer's encoder and decoder are both conditioned on visual observations and executed actions. This forces the resulting discrete 16-bit code to represent only the contact dynamics that vision and actions cannot explain.
  2. TacInno-VLA Policy: The researchers integrated a lightweight tactile branch into a pre-trained VLA backbone. During training, the policy is given an auxiliary task: predict the "innovation code" generated by the tokenizer for the current step.
  3. Deployment Efficiency: Crucially, the tokenizer and the code-prediction heads are only used during training. At deployment, the robot uses its learned tactile representations to predict actions directly, adding zero computational overhead.
The TacInno tokenizer architecture, showing how visual and action conditioning isolates innovation.

Datasets

The authors evaluated TacInno across both simulated and physical environments:

  • UniVTAC Benchmark: Six simulated contact-rich tasks (e.g., Insert Hole, Pull-out Key, Lift Bottle) using the Isaac Sim environment.
  • Real-World Tasks: Four challenging tasks—wiping a board, inserting a plug, lifting a can, and unscrewing a bottle cap—performed by a Franka FR3 robot equipped with a Xense tactile sensor.

Key Insights and Ideas

  • The 97% Redundancy: The authors found that in a standard setup, context (vision and actions) already accounts for 97% of the information in future tactile changes. By focusing on the remaining 3% of "innovation," the model learns much more effectively.
  • True Tactile Reliance: To prove their model wasn't just "cheating" with vision, the authors performed a tactile-off test. When tactile input was removed at test time, TacInno-VLA's success rate on a lifting task plummeted from 88% to 0%, whereas baseline models barely changed. This confirms that TacInno-VLA actually listens to its fingers.
  • Timing is Everything: Innovation peaks at critical moments, such as immediately before contact or at the onset of an incipient slip.
Temporal distribution of tactile innovation during a can-lifting task.

Key Results

TacInno-VLA consistently outperformed state-of-the-art visuo-tactile baselines:

  • Simulation: Achieved an average success rate of 87.5%, outperforming the strongest baseline (SimpleTouch) by 10.7 percentage points.
  • Real World: Achieved an average success rate of 70.5%, a 9.0 percentage point improvement over baselines.
  • Robustness: It was the only model to maintain at least a 75% success rate across every simulated task, whereas baselines often failed completely on specific tasks.
Success rates in simulation and real-world robot tasks.

Limitations and Future Work

While TacInno is a significant step forward, the authors note a few areas for growth:

  • Demonstration Dependency: The tokenizer currently requires high-quality expert demonstrations to learn what is "predictable."
  • Generalization to Other Senses: The concept of "innovation" is powerful and could potentially be applied to other redundant modalities, such as audio or proprioception.
  • Self-Supervision: Future work could explore learning these innovation codes from unlabeled robot interaction data, reducing the need for human experts.

Summary updated

Abstract

Vision-Language-Action (VLA) models have recently been equipped with tactile sensing to perceive visually subtle contact events in contact-rich manipulation. However, adding tactile input yields inconsistent gains, and some visuo-tactile VLA models rely little on tactile input, because much of the tactile change, such as the gel deformation during gripper closure, is predictable from visual observations and executed actions. To address this problem, we introduce TacInno, which isolates the contact dynamics that vision and executed actions cannot explain, termed tactile innovation, and uses tactile innovation only as a training target. Specifically, the TacInno tokenizer, conditioned on visual observations and executed actions, compresses the future tactile change into a discrete code, and the policy, TacInno-VLA, learns to predict the code, adding no module for the code at deployment. TacInno-VLA achieves average success rates of 87.5% across six simulation tasks and 70.5% across four real-world tasks, outperforming the strongest visuo-tactile baseline by 10.7 and 9.0 percentage points, respectively. Ablation studies validate the contributions of visual and action conditioning, multi-frame encoding, and auxiliary code prediction. Together, these results demonstrate the effectiveness of tactile innovation as a supervision signal for contact-rich manipulation.
BibTeX
@misc{li2026tacinno,
  author = {Weile Li and Yunfan Lou and Muqing Shi and Zihao Zhang and Zhenbiao Huang and Boxian Yao and Xiyu Zhu and Zhibo Pang},
  title = {{TacInno: Teaching Vision-Language-Action Models the Contact Dynamics Beyond Vision and Action}},
  year = {2026},
  eprint = {2611.26405},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26405},
}
Figure