TL;DR
3DTouch is a novel framework that enhances Vision-Language-Action (VLA) models for dexterous manipulation by integrating spatially grounded tactile feedback and explicit 3D geometric relations. By aligning multi-finger tactile forces and object surfaces in a shared 3D coordinate frame, it achieves a 65% average success rate across challenging real-world tasks, outperforming standard VLA baselines by 42 percentage points.
What Problem Does This Paper Address?
Dexterous manipulation—using multi-fingered hands to perform complex tasks—is notoriously difficult for robots. Two major hurdles stand in the way: occlusion and visual ambiguity.
When a robot hand grasps an object, its fingers often hide the very contact points the robot needs to see to adjust its grip. Furthermore, two visually identical scenes might represent completely different physical states; for instance, a funnel might look perfectly aligned with a flask from a camera's perspective but actually be resting on the rim.
Current VLA models primarily rely on 2D visual inputs, which struggle to capture these fine-grained 3D spatial relationships and local contact forces. While some models incorporate tactile data, they often treat it as a separate "bag of numbers" rather than grounding it in the physical 3D space where the manipulation is happening.
Methods
The authors propose 3DTouch, which augments a pretrained VLA (specifically the flow-matching based $\pi_0$ model) with two key modules:
- 3D Tactile Grounding (3DTG): Instead of just feeding raw force values, 3DTG uses the robot's forward kinematics to map the exact 3D position and force vector of every "taxel" (tactile pixel) on all five fingers into the robot's base coordinate frame. This aligns tactile data perfectly with the RGB-D scene point cloud, allowing the model to "feel" exactly where contact is occurring relative to the environment.
- Geometric Relation Modeling (GRM): This module explicitly calculates the relative distances and displacements between hand keypoints and object surfaces (Hand-Object relations) and between interacting objects (Object-Object relations). It uses a lightweight pose estimator to align pre-reconstructed object meshes with the live scene, providing the "Action Expert" with clear cues for alignment and insertion.
These 3D representations are converted into tokens and concatenated with standard vision and language tokens, conditioning the VLA's action generation on precise spatial and physical context.
Datasets
The researchers evaluated 3DTouch on a custom dataset of approximately 500 real-robot demonstrations (100 per task). The evaluation platform consisted of a UR7e robot arm equipped with an XHand—a dexterous hand featuring 600 taxels (120 per finger) capable of 3D force sensing.
The five tasks chosen represent classic "contact-rich" challenges:
- Funnel Insertion: High-precision alignment.
- Plug Assembly: Requires sensing resistance and adjusting.
- Button Press: Requires localized force application.
- Sponge Wiping: Involves sustained contact over a surface.
- Pipette Press: Complex multi-stage manipulation.
Key Insights and Ideas
- Spatial Grounding is Non-Negotiable: One of the most striking findings was that representing tactile data in the robot-base frame (Global) performed significantly better than in the hand-local frame. When the model knows where a force is relative to the target object, it can make much more effective corrections.
- Explicit Relations Beat Implicit Learning: Even with high-quality point clouds, deep learning models often struggle to infer tiny clearances (like the gap between a plug and a socket). Explicitly calculating these distances via the GRM module provided a "shortcut" that significantly boosted success in insertion tasks.
- Complementary Modalities: Ablation studies showed that while point clouds or tactile data alone help, their combination is synergistic. Tactile feedback helps refine contact after the hand reaches the object, while geometric relations guide the approach and alignment.
Key Results
3DTouch demonstrated a massive leap in performance over existing methods:
- Average Success Rate: 65.0%, compared to 23.0% for the $\pi_0$ baseline and 45.0% for the strongest tactile-aware VLA (Tactile-VLA).
- Task-Specific Gains: In the difficult Plug Assembly task, 3DTouch achieved a 60% success rate, whereas the standard VLA baseline failed completely (0%).
- Robustness: 3DTouch proved much more resilient to "target pose variations." When objects were slightly shifted or rotated from their training positions, 3DTouch's success rate dropped by only 5%, compared to a 13.3% drop for tactile-augmented baselines.
Limitations and Future Work
The authors acknowledge a few key limitations:
- Reliance on Object Priors: The current GRM module requires pre-reconstructed object meshes and a pose estimator. This limits the system's ability to handle completely novel, unseen objects on the fly.
- Inference Speed: While the 3D representations are computed once per observation, the added complexity of processing point clouds and relations adds computational overhead.
- Future Directions: The team aims to reduce reliance on reference geometry and explore how 3DTouch can scale to diverse robot embodiments beyond the UR7e/XHand setup.
For researchers in robotics and AI, 3DTouch serves as a compelling argument that for the "last centimeter" of manipulation, we need to move beyond 2D pixels and embrace grounded, relational 3D physics.
Summary updated