TaF-Foundation: Wrench-Grounded Cross-Sensor Tactile Representations

Jiaming Jiang, Yuzhe Huang, Baokun Yang, Hao Liang, Li Gao, Haoran Guo, Pei Lin, Shengcheng Luo, Chenxi Xiao, Wanlin Li, Ziyuan JiaoarXiv 2026 · arXiv

TL;DR

TaF-Foundation is a large-scale framework that creates a "universal language" for tactile sensing by grounding diverse sensor data in 6D wrenches (force and torque). By training on a massive new dataset of 100 million tactile-wrench pairs, the authors enable cross-hardware transfer and significantly boost the success rates of robots performing complex, contact-rich manipulation tasks.

What Problem Does This Paper Address?

In computer vision, we have ImageNet; in robotics, we are still searching for the "Touch-Net." The primary hurdle is sensor heterogeneity. Tactile sensors come in wildly different flavors: some are cameras looking at deforming rubber (optical), some measure magnetic field changes, and others use pressure-sensitive matrices. Each produces a different data format (images, vectors, or grids), making it nearly impossible to train a model on one sensor and use it on another.

Current methods often try to align these sensors by having them touch the same objects, but this doesn't scale. The authors of TaF-Foundation argue that we should stop trying to map sensors to each other and instead map them all to physics. Because every tactile interaction obeys Newtonian mechanics, a 6D wrench (3D force + 3D torque) serves as a universal physical reference that remains constant regardless of which sensor is measuring it.

Methods

The authors developed a unified architecture designed to ingest any tactile data and project it into a shared "physically grounded" feature space.

  1. Sensor-Specific Adapters: To handle the "alphabet soup" of data formats, they use custom front-ends. Optical data is processed via patch-based projections, while vector and matrix data use MLPs and CNNs, respectively.
  2. Shared Transformer Encoder: All tokens are fed into a shared Transformer (initialized from DINOv2). A learnable "physical query" token aggregates these features into a final 768-dimensional representation.
  3. Wrench-Grounded Learning: The model is trained with a multi-task objective:
  4. Wrench Reconstruction: A decoder predicts the 6D wrench from the tactile embedding.
  5. Representation Alignment: The tactile embedding is aligned with an embedding from a dedicated wrench encoder (using an Exponential Moving Average, or EMA, target for stability).
  6. TaF-Policy: For downstream tasks, the frozen tactile encoder is integrated into an Action Chunking Transformer (ACT) policy alongside visual inputs and robot states to generate manipulation commands.

Datasets

The scale of the data introduced here is a major contribution:

  • TaF-Dataset: Over 100 million paired frames of tactile observations and wrench measurements.
  • Hardware Diversity: Data was collected from 24 different sensors across four sensing families (Optical, Magnetic, Pressure, and Piezoresistive).
  • Interaction Diversity: The dataset includes over 1,445 unique probe-sensor pairs and 200+ probe geometries, covering actions like pressing, sliding, and rolling.
  • Benchmarks: The authors evaluate against established tactile models like Sparsh, T3, and AnyTouch 2.

Key Insights and Ideas

  • Physics as the Universal Bridge: You don't need to collect data where two different sensors touch the same object simultaneously. As long as each sensor is paired with a high-quality force/torque (wrench) sensor during data collection, the model can learn a shared representation.
  • The Power of 6D: Most previous work focused only on 3D force. By including 3D torque, the model captures rotational dynamics that are critical for tasks like inserting a key or removing a power bank.
  • Frozen Backbones Work: Once grounded in physics, the tactile encoder becomes a robust "feature extractor." The authors found that freezing this backbone during policy training prevents the robot from "forgetting" physical constraints while learning specific manipulation skills.

Key Results

The quantitative gains are substantial across the board:

  • Wrench Estimation: The model reduced single-frame force error (RMSE) by 17.1% compared to the strongest baseline.
  • Cross-Sensor Retrieval: When asked to find a similar "touch" experience from a completely different sensor type, TaF-Foundation improved the R@1 accuracy by 13.11 percentage points.
  • Simulated Manipulation: On the UniVTAC benchmark, TaF-Policy achieved a 51.33% average success rate (vs. 47.33% for the best baseline).
  • Real-World Manipulation: In real-world tests (bottle lifting, key insertion, etc.), the gap widened significantly. TaF-Policy achieved a 46.25% success rate, nearly doubling the performance of the strongest baseline (23.75%).

Limitations and Future Work

Despite the progress, the authors note several areas for improvement:

  • Few-Shot Requirement: While the model generalizes well, adapting to a brand-new, unseen sensor still requires a small amount of labeled tactile-wrench data.
  • Precision Issues: Tasks requiring sub-millimeter precision, such as perfect key-to-keyhole alignment, still see failures due to the inherent resolution limits of some sensors or the policy's representation.
  • Grasp Stability: The system can still struggle with dynamic slippage during high-speed movements.
  • Future Directions: The team aims to reduce the need for sensor-specific labeled data even further, potentially moving toward zero-shot adaptation for new hardware.

For more details, visit the Project Page.

Summary updated

Abstract

Tactile sensing is essential for contact-rich manipulation, yet heterogeneous sensing mechanisms and observation formats hinder cross-hardware transfer. Since physical contact is governed by Newtonian mechanics, we ground tactile representations in six-dimensional wrenches, establishing a common physical reference across sensing mechanisms. Specifically, we pair tactile observations from all sensors with wrench measurements, enabling shared physical supervision without requiring direct tactile-to-tactile pairing between sensors. To support learning at scale, we develop a dedicated acquisition platform and collect over 100 million paired tactile observations and wrench measurements from 24 sensors spanning four families of sensing principles. Building on this dataset, our framework combines sensor-specific input adaptation with a shared tactile encoder. By learning to reconstruct wrenches, the encoder maps observations from different sensors into a shared, continuous feature space grounded in contact physics. To leverage these representations for manipulation, we introduce TaF-Policy, which integrates tactile features with visual observations and robot states to generate action chunks. We evaluate the learned representations through wrench estimation, cross-sensor retrieval, few-shot adaptation to unseen sensors, and robotic manipulation. On the core benchmark, our method reduces single-frame force RMSE by 17.1% and improves cross-sensor retrieval R@1 by 13.11 percentage points over the strongest baseline. In manipulation experiments, TaF-Policy achieves average success rates of 51.33% across three simulated tasks and 46.25% across four real-world tasks, compared with 47.33% and 23.75%, respectively, for the strongest baselines evaluated on the same task sets. Project page: https://mrjiangjm.github.io/TaF-Foundation/.
BibTeX
@misc{jiang2026taf,
  author = {Jiaming Jiang and Yuzhe Huang and Baokun Yang and Hao Liang and Li Gao and Haoran Guo and Pei Lin and Shengcheng Luo and Chenxi Xiao and Wanlin Li and Ziyuan Jiao},
  title = {{TaF-Foundation: Wrench-Grounded Cross-Sensor Tactile Representations}},
  year = {2026},
  eprint = {2611.20718},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20718},
}
Figure