Language-Grounded Force/Torque Pretraining for Extrinsic Contact Perception

Kazuki Takahashi, Hiroaki Murakami, Ryota Sakuma, Mitsuhiro Kamezaki, Matthew Ishige, Yoshihiro KawaharaarXiv 2026 · arXiv

TL;DR

This paper introduces a method for pretraining robotic force/torque (F/T) sensors by contrastively aligning their signal histories with a frozen language embedding space. By training on massive simulated pick-and-place data, the authors create a semantically structured representation that allows robots to recognize physical states—like object weight and contact events—on unseen objects and efficiently adapt to new real-world tasks with minimal labeled data.

What Problem Does This Paper Address?

Fine manipulation requires robots to understand "extrinsic contact"—the moment a grasped object touches the environment (e.g., a cup hitting a table). While vision and tactile sensors help, they often miss these subtle interactions due to occlusions or limited local geometry.

Six-axis force/torque (F/T) sensors at the robot's wrist or fingertips provide the necessary data, but these signals are notoriously difficult to interpret. A single F/T reading is a messy composite of gravity, inertia, the robot's own motion, and the actual contact force. Historically, it has been unclear if we can build a reusable representation of these signals that generalizes across different objects, grasp configurations, and environments.

Methods

The authors propose Language-Grounded F/T Pretraining. The core idea is to treat F/T histories as a "modality" that can be aligned with natural language, similar to how CLIP aligns images with text.

  1. Automated Simulation: They used the Genesis physics engine to generate over 1.5 million windows of F/T data from 40,000+ pick-and-place episodes. Because it's a simulation, they have "privileged" access to the ground truth (e.g., "the object is slipping," "the surface is wobbling").
  2. Contrastive Learning: They trained an F/T encoder (evaluating PatchTST, CNN, and MLP architectures) to map 0.8-second signal histories into a shared embedding space with a frozen Sentence-BERT text encoder.
  3. Zero-Shot Readout: At inference time, the robot can "query" its physical state by comparing the current F/T embedding to text prototypes like "Placing an object" or "Holding a heavy object" using cosine similarity.
  4. Real-World Transfer: To bridge the sim-to-real gap, they used a simple unlabeled calibration step to normalize real sensor data to the simulated distribution, keeping the pretrained encoder weights frozen.
  5. Adaptation: For tasks not seen during pretraining (like connector insertion), they used LoRA (Low-Rank Adaptation) to efficiently fine-tune the representation with very few real-world examples.
Overview of the automated dataset construction, contrastive learning, and evaluation phases.

Datasets

  • Pretraining (Sim): 437,244 windows from 40,851 episodes involving 892 Google Scanned Objects (GSO).
  • Evaluation (Sim): 111,355 windows involving 75 YCB objects (completely unseen during training).
  • Real-World Placement: 80 trials across different objects (wooden cuboid, cork cylinder, compliant ball) and support conditions (fixed vs. wobbling).
  • Connector Insertion: A custom real-robot dataset of 125 successful episodes for evaluating few-shot adaptation.

Key Insights and Ideas

  • Language as a Semantic Anchor: Aligning F/T signals with language doesn't just make them "queryable"—it provides a structured prior. The authors found that language-prototype readout actually outperformed standard multi-task supervised learning on unseen objects, suggesting that the semantic structure of language helps the model generalize physical concepts.
  • Beyond Simple Thresholds: Traditional contact detection often relies on simple rules (e.g., "stop when vertical force reverses"). This fails on compliant objects (like a soft ball) or moving surfaces. The pretrained model, however, learns to recognize the dynamics of load transfer, succeeding where rule-based baselines fail.
  • Adaptable Initialization: For entirely new events (like a connector becoming "fully seated"), the pretrained representation is most valuable as a starting point for adaptation (LoRA) rather than a fixed feature extractor.
Example force histories showing how different physical conditions (weight, slip, support type) manifest in the signal.

Key Results

  • Generalization: In simulation, the model achieved high Macro F1 scores (>93%) for recognizing actions and object weights on objects it had never seen before.
  • Real-World Superiority: In real-world contact detection, the proposed method achieved an 83.3% success rate across various objects, significantly outperforming a vertical-force rule-based baseline (66.7%). Notably, the baseline had a 0% success rate on compliant balls and wobbling supports, while the pretrained model remained functional.
  • Data Efficiency: For the new task of connector insertion, using LoRA adaptation on the pretrained model with just one training episode reached ~91.7% F1, whereas training from scratch only reached ~77.5%.

Limitations and Future Work

  • Task Diversity: The pretraining was limited to pick-and-place. It doesn't yet cover more complex "contact-rich" maneuvers like sustained sliding, rotation, or manipulating deformable objects.
  • Calibration: The system still requires a brief unlabeled calibration phase to handle the specific offsets and scales of real-world sensors.
  • Failure Modes: The current evaluation focused on successful episodes; the authors note that future work needs to address failure discrimination and recovery.
  • Language Robustness: The study used a finite set of predefined text templates and did not test for robustness against varied phrasing or paraphrasing.

Summary updated

Abstract

Fine manipulation requires robots to recognize when a grasped object makes or changes contact with its environment, yet such extrinsic contact is not always directly observable from vision or local contact geometry. Force and torque signals carry this information, but also vary with object weight, grasp configuration, and robot motion, leaving it unclear whether they can provide a reusable representation of physical state. Here we show that bilateral six-axis force/torque histories can be pretrained as a semantically structured physical representation by aligning them with a frozen language space using supervision generated from simulated pick-and-place interactions. On unseen objects in simulation, the learned representation supports text-prototype readout of robot action, object weight, and grasp interaction. In real-world placement, unlabeled calibration enables contact and support-state recognition without updating the encoder, outperforming a vertical-force sign-reversal rule baseline. For connector insertion, an event absent from pretraining, low-rank adaptation improves data-efficient learning over training from scratch. These results show that force/torque histories can support both direct readout of known physical states and efficient adaptation to new extrinsic-contact events.
BibTeX
@misc{takahashi2026language,
  author = {Kazuki Takahashi and Hiroaki Murakami and Ryota Sakuma and Mitsuhiro Kamezaki and Matthew Ishige and Yoshihiro Kawahara},
  title = {{Language-Grounded Force/Torque Pretraining for Extrinsic Contact Perception}},
  year = {2026},
  eprint = {2610.39314},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39314},
}
Figure