3DTouch: A Tactile-aware VLA Model with 3D Geometric Relations for Dexterous Manipulation

Xin Zhang, Shiqi Zhang, Siyuan Cao, Juncheng Cai, Yedong Shen, Yuxuan Gao, Dianlong Wen, Kaixue Long, Lu Zhang, Yao Li, Jiajun Deng, Yanyong ZhangarXiv 2026 · arXiv

TL;DR

3DTouch is a novel framework that enhances Vision-Language-Action (VLA) models for dexterous manipulation by integrating spatially grounded tactile feedback and explicit 3D geometric relations. By aligning multi-finger tactile forces and object surfaces in a shared 3D coordinate frame, it achieves a 65% average success rate across challenging real-world tasks, outperforming standard VLA baselines by 42 percentage points.

What Problem Does This Paper Address?

Dexterous manipulation—using multi-fingered hands to perform complex tasks—is notoriously difficult for robots. Two major hurdles stand in the way: occlusion and visual ambiguity.

When a robot hand grasps an object, its fingers often hide the very contact points the robot needs to see to adjust its grip. Furthermore, two visually identical scenes might represent completely different physical states; for instance, a funnel might look perfectly aligned with a flask from a camera's perspective but actually be resting on the rim.

Current VLA models primarily rely on 2D visual inputs, which struggle to capture these fine-grained 3D spatial relationships and local contact forces. While some models incorporate tactile data, they often treat it as a separate "bag of numbers" rather than grounding it in the physical 3D space where the manipulation is happening.

Methods

The authors propose 3DTouch, which augments a pretrained VLA (specifically the flow-matching based $\pi_0$ model) with two key modules:

  1. 3D Tactile Grounding (3DTG): Instead of just feeding raw force values, 3DTG uses the robot's forward kinematics to map the exact 3D position and force vector of every "taxel" (tactile pixel) on all five fingers into the robot's base coordinate frame. This aligns tactile data perfectly with the RGB-D scene point cloud, allowing the model to "feel" exactly where contact is occurring relative to the environment.
  2. Geometric Relation Modeling (GRM): This module explicitly calculates the relative distances and displacements between hand keypoints and object surfaces (Hand-Object relations) and between interacting objects (Object-Object relations). It uses a lightweight pose estimator to align pre-reconstructed object meshes with the live scene, providing the "Action Expert" with clear cues for alignment and insertion.
Figure 2. Pipeline of 3DTouch. (a) 3D Tactile Grounding (3DTG) back-projects RGB-D observations into scene point clouds and transforms taxel positions and force vectors into a shared robot-base frame using forward kinematics (FK). Separate encoders produce scene tokens and finger-level tactile tokens. (b) Geometric Relation Modeling (GRM) aligns reference object meshes using estimated poses and encodes relative displacements and distances between hand landmarks and nearby object surface points, and between representative regions of interacting objects, into hand–object (HO) and object–object (OO) tokens. The scene, tactile, and relation tokens jointly condition the action expert, while the original vision-language pathway and flow-matching objective are preserved.

These 3D representations are converted into tokens and concatenated with standard vision and language tokens, conditioning the VLA's action generation on precise spatial and physical context.

Datasets

The researchers evaluated 3DTouch on a custom dataset of approximately 500 real-robot demonstrations (100 per task). The evaluation platform consisted of a UR7e robot arm equipped with an XHand—a dexterous hand featuring 600 taxels (120 per finger) capable of 3D force sensing.

The five tasks chosen represent classic "contact-rich" challenges:

  • Funnel Insertion: High-precision alignment.
  • Plug Assembly: Requires sensing resistance and adjusting.
  • Button Press: Requires localized force application.
  • Sponge Wiping: Involves sustained contact over a surface.
  • Pipette Press: Complex multi-stage manipulation.
Figure 4. Overview of five contact-rich manipulation tasks performed on the UR7e robot equipped with an XHand dexterous hand. Each column shows the start and end states of a task, with red dashed arrows indicating the intended manipulation motions.

Key Insights and Ideas

  • Spatial Grounding is Non-Negotiable: One of the most striking findings was that representing tactile data in the robot-base frame (Global) performed significantly better than in the hand-local frame. When the model knows where a force is relative to the target object, it can make much more effective corrections.
  • Explicit Relations Beat Implicit Learning: Even with high-quality point clouds, deep learning models often struggle to infer tiny clearances (like the gap between a plug and a socket). Explicitly calculating these distances via the GRM module provided a "shortcut" that significantly boosted success in insertion tasks.
  • Complementary Modalities: Ablation studies showed that while point clouds or tactile data alone help, their combination is synergistic. Tactile feedback helps refine contact after the hand reaches the object, while geometric relations guide the approach and alignment.

Key Results

3DTouch demonstrated a massive leap in performance over existing methods:

  • Average Success Rate: 65.0%, compared to 23.0% for the $\pi_0$ baseline and 45.0% for the strongest tactile-aware VLA (Tactile-VLA).
  • Task-Specific Gains: In the difficult Plug Assembly task, 3DTouch achieved a 60% success rate, whereas the standard VLA baseline failed completely (0%).
  • Robustness: 3DTouch proved much more resilient to "target pose variations." When objects were slightly shifted or rotated from their training positions, 3DTouch's success rate dropped by only 5%, compared to a 13.3% drop for tactile-augmented baselines.

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Reliance on Object Priors: The current GRM module requires pre-reconstructed object meshes and a pose estimator. This limits the system's ability to handle completely novel, unseen objects on the fly.
  • Inference Speed: While the 3D representations are computed once per observation, the added complexity of processing point clouds and relations adds computational overhead.
  • Future Directions: The team aims to reduce reliance on reference geometry and explore how 3DTouch can scale to diverse robot embodiments beyond the UR7e/XHand setup.

For researchers in robotics and AI, 3DTouch serves as a compelling argument that for the "last centimeter" of manipulation, we need to move beyond 2D pixels and embrace grounded, relational 3D physics.

Summary updated

Abstract

In contact-rich dexterous manipulation, occlusion by the fingers and interacting objects obscures local geometry, preventing visual observations from fully capturing contact locations. Visually similar scenes may also differ subtly in the relative positions of interacting objects, leading to different contact states. We propose 3DTouch, a tactile-geometric conditioning approach for vision-language-action (VLA) models. 3DTouch constructs a shared 3D visuo-tactile representation by expressing RGB-D scene points and dense five-finger tactile measurements in the robot-base frame, preserving each taxel's spatial location and force vector. Hand-object and object-object relations derived from hand landmarks and pose-aligned reconstructed surfaces explicitly describe the relative arrangement of interacting entities. Together, these representations condition the VLA action expert on spatially localized contact forces and relative surface geometry, providing explicit cues for contact adjustment and object alignment. We evaluate 3DTouch on five real-world contact-rich dexterous manipulation tasks using a UR7e robot arm equipped with an XHand dexterous hand. 3DTouch improves the average success rate by 42 percentage points over the corresponding VLA baseline and by 20 percentage points over the strongest evaluated alternative. Ablation studies reveal the complementary benefits of scene geometry and spatial tactile feedback, with further gains from shared spatial grounding and explicit geometric relations.
BibTeX
@misc{zhang2026dtouch,
  author = {Xin Zhang and Shiqi Zhang and Siyuan Cao and Juncheng Cai and Yedong Shen and Yuxuan Gao and Dianlong Wen and Kaixue Long and Lu Zhang and Yao Li and Jiajun Deng and Yanyong Zhang},
  title = {{3DTouch: A Tactile-aware VLA Model with 3D Geometric Relations for Dexterous Manipulation}},
  year = {2026},
  eprint = {2611.23961},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23961},
}
Figure