TL;DR
This paper introduces a novel framework for learning dexterous robotic manipulation policies by effectively leveraging 3D tactile data. By proposing a canonical representation that aligns distributed sensor coordinates and a self-supervised pretraining strategy based on local and net force predictions, the authors achieve a 78% average success rate across contact-rich, fine-grained real-world tasks, significantly outperforming existing 2D image-based and un-pretrained graph baselines.
What Problem Does This Paper Address?
Dexterous robotic hands equipped with distributed tactile sensors generate high-dimensional, sparse 3D data. Unlike computer vision, where massive datasets and powerful pretrained models (like DINO or CLIP) are readily available, tactile sensing lacks standardized datasets and universal backbones.
To bypass this, many current methods flatten tactile data into vectors or project them into 2D images to utilize standard visual backbones. However, this projection distorts or destroys the inherent 3D spatial relationships between taxels (tactile pixels) across different fingers. Conversely, treating the data natively as a 3D graph often fails in practice because Graph Neural Networks (GNNs) require massive amounts of task-specific data to learn meaningful representations from scratch. This paper addresses the challenge of efficiently learning spatial and force features directly from native 3D tactile data for complex, contact-rich manipulation tasks.
Methods
The authors present a two-part solution: a Canonical Tactile Representation and a Force-Based Pretraining pipeline.
[Raw 3D Tactile Data] -> [Canonicalization (Unit Frame)] -> [GraphMAE Encoder] -> [Force-Based Pretraining (Local + Net)] -> [Visuo-Tactile Diffusion Policy]
- Canonical Tactile Representation: Instead of using raw 3D coordinates, which vary wildly across different finger segments, the authors normalize the coordinates of the taxels within each individual sensor into a unified unit frame (ranging from -1 to 1). To preserve global context, they append the 6D pose of each sensor's origin relative to the hand's base ($P^s$). Combined with the 3D force vector ($F$), each taxel is represented as a 12D feature vector: $R = [P^s, T, F]$. This representation is then processed using a Graph Neural Network based on a 4-neighborhood connectivity graph.
- Force-Based Pretraining: To teach the GNN encoder useful manipulation features before task training, the authors implement two self-supervised objectives using collected "play data":
- Masked Local Force Prediction: Randomly masks out force readings on certain taxels and forces the GNN to reconstruct them based on neighboring taxels.
- Net Force Prediction: Uses the encoder's latent features to predict the total global net force ($F^n_G$) exerted across the hand, ensuring the network understands how local forces scale up to macro-movements.
Finally, this pretrained tactile encoder is frozen/fine-tuned within a visuo-tactile Diffusion Policy framework, which combines the tactile features with visual features from a DinoV2 backbone and the robot's proprioceptive state to output actions.
Datasets
The authors collected their own datasets using a real-world platform consisting of a JAKA MiniCobo arm and a 16-DoF LEAP Hand equipped with PaXini distributed tactile sensors.
- Tactile Play Data: Unstructured interaction data used exclusively for the self-supervised pretraining phase.
- Expert Demonstrations: 100 human-teleoperated demonstrations per task for four distinct evaluation benchmarks: Open Box, Reorientation, Flip, and Assembly.
Key Insights and Ideas
- The Power of Canonicalization: Raw taxel coordinates within a sensor pad are separated by mere millimeters. GNNs struggle to differentiate these tiny spatial variances. Canonicalizing them to a $[-1, 1]$ unit frame amplifies their relative distances, making local features highly distinguishable to the network while standardizing representations across different fingers.
- Multi-Scale Force Awareness: Manipulation requires understanding both micro-interactions (is an object slipping from a specific finger pad?) and macro-dynamics (what is the total force being applied to the object?). By combining masked local force prediction with net force prediction, the pretraining task forces the encoder to bridge this multi-scale gap.
- Functional Division of Labor: Ablation studies revealed that spatial information ($P^s$ and $T$) is primarily responsible for guiding gross hand poses, whereas force information ($F$) drives the fine-grained adjustments necessary to maintain stable grasps during high-occlusion tasks.
Key Results
The proposed method was evaluated against several baselines, including a pure vision Diffusion Policy (DP), an MLP-flattened tactile baseline (HATO), an image-projected tactile baseline (T-DEX), and a standard un-pretrained GNN.
| Method | Open Box | Reorientation | Flip | Assembly | Avg |
|---|---|---|---|---|---|
| DP (Vision Only) | 90% | 60% | 20% | 40% | 53% |
| HATO (MLP) | 70% | 60% | 10% | 50% | 48% |
| T-DEX (2D Image) | 80% | 70% | 40% | 60% | 63% |
| GNN (Un-pretrained) | 0% | 0% | 0% | 0% | 0% |
| Ours | 90% | 70% | 80% | 70% | 78% |
The un-pretrained GNN completely failed (0% success), causing the hand to shake violently due to the difficulty of learning raw 3D spatial-force correlations from scratch. The proposed method achieved a 78% average success rate, showing massive improvements in the highly complex Flip (80%) and Assembly (70%) tasks where visual occlusion is severe. Furthermore, the policy successfully generalized to unseen objects with varying geometries and friction properties, achieving a 5/8 success rate on box opening and 6/8 on cap flipping.
Limitations and Future Work
While the framework demonstrates strong robustness, the authors note that the policy still exhibits limited generalization when encountering objects with drastically different shapes or dynamics than those seen during training. They point toward incorporating online, rapid tactile adaptation mechanisms as a promising future direction to overcome this limitation.
Summary updated