TL;DR
CoRA-VTLA enhances pretrained Vision-Language-Action (VLA) models with tactile sensing to master precision manipulation tasks like peg-in-hole insertion. By introducing a queryable memory of recent contact responses and a "gaze loss" that grounds tactile tokens in visual contact regions, it achieves a 73.33% success rate in simulation—significantly outperforming standard tactile fusion methods.
What Problem Does This Paper Address?
While large-scale VLA models have shown impressive general-purpose capabilities, they often struggle with "precision manipulation"—tasks where sub-millimeter errors lead to jamming or part deformation. Visual feedback is insufficient here because a camera cannot distinguish between a robot arm moving freely and one that is pushing against a hidden obstruction.
Tactile sensing provides the necessary feedback, but integrating it into pretrained VLAs presents two major hurdles:
- Temporal Ambiguity: A single snapshot of tactile data is ambiguous. Is the current force due to initial contact or a sustained jam? To decide on a corrective action, the model needs to know how the contact has evolved in response to recent movements.
- Attention Dispersion: Simply "injecting" tactile data into a VLA can distract the model's visual attention, causing it to lose focus on the critical regions where contact is actually occurring.
Methods
The authors propose CoRA-VTLA, which augments a pretrained VLA (specifically $\pi_{0.5}$) with lightweight tactile modules through a two-stage training process.
Key Components:
- Queryable Action-Response Memory: Instead of a simple history buffer, this module uses a temporal Transformer to encode recent tactile events (changes in pressure and force/torque). The action expert then "queries" this memory based on the current robot state to retrieve the most relevant past responses for the next move.
- Tactile Semantic Token ($S$): A new token is added to the VLM's input prefix. This token is trained to predict high-level contact properties like stability and load changes.
- Contact-Guided Gaze Supervision: To prevent attention drift, the authors introduce a "gaze loss." They project the robot's fingertip contact points into the camera's image space and penalize the model if the tactile semantic token's attention doesn't overlap with these projected regions.
- Two-Stage Training:
- Response Pretraining: Tactile encoders are first trained to predict the consequences of actions (e.g., "if I move down, how will pressure change?").
- Joint Post-Training: The entire system is fine-tuned using a combination of action loss, semantic classification loss, and the novel gaze loss.
Datasets
The model was evaluated on a diverse set of benchmarks:
- Custom MuJoCo Tasks: Nail insertion, plug insertion, and knob operation.
- UniVTAC Benchmark: A public benchmark for optical-tactile sensing, specifically the "Insert Tube" task.
- Real-Robot Tasks: Soft paper cup stacking (requiring delicate force control), stamping on paper, and test tube insertion into a rack.
Key Insights and Ideas
- History is the Key to Recovery: The "aha moment" here is that the relationship between an action and the resulting tactile change is more informative than the raw force value. Figure 4 illustrates this: when a plug jams, the memory of the force rising during the downward move allows the model to trigger a "back off and realign" sequence.
- Spatial Grounding Prevents Distraction: By forcing the model to "look" at the fingertips when processing tactile tokens, the researchers found they could maintain the VLA's visual focus on the task. Without this gaze supervision, the model's attention often dispersed, leading to lower success rates.
- Cross-Modal Generalization: The architecture isn't tied to a specific sensor. The same memory design worked for both numeric pressure arrays (MuJoCo/Real Robot) and optical tactile images (UniVTAC), suggesting a robust general-purpose tactile interface for VLAs.
Key Results
- Simulation Superiority: In MuJoCo tasks, CoRA-VTLA achieved a 73.33% mean success rate, compared to just 50% for "Direct Fusion" (simply concatenating tactile data) and 31.33% for a vision-only baseline.
- Real-World Robustness: On a physical UR5e robot, the system averaged 66.7% success across three challenging tasks. It proved particularly adept at recovering from "rim contact" during tube insertion—a scenario where vision-only models typically fail.
- Attention Accuracy: Gaze supervision increased the semantic token's attention share on the actual contact region from a measly 2.77% to 46.65%.
Limitations and Future Work
The authors identify a few areas for improvement:
- Memory Horizon: The current memory window is 0.8 seconds. Longer-term memory might be needed for tasks with complex, multi-stage contact sequences.
- Safety: While the model can recover from jams, it doesn't explicitly optimize for safety (e.g., minimizing peak impact forces).
- Scaling: Future work could explore training on larger, more diverse tactile datasets to see if the "tactile grounding" benefits scale as impressively as visual grounding has for VLAs.
Summary updated