TL;DR
This paper introduces a method for pretraining robotic force/torque (F/T) sensors by contrastively aligning their signal histories with a frozen language embedding space. By training on massive simulated pick-and-place data, the authors create a semantically structured representation that allows robots to recognize physical states—like object weight and contact events—on unseen objects and efficiently adapt to new real-world tasks with minimal labeled data.
What Problem Does This Paper Address?
Fine manipulation requires robots to understand "extrinsic contact"—the moment a grasped object touches the environment (e.g., a cup hitting a table). While vision and tactile sensors help, they often miss these subtle interactions due to occlusions or limited local geometry.
Six-axis force/torque (F/T) sensors at the robot's wrist or fingertips provide the necessary data, but these signals are notoriously difficult to interpret. A single F/T reading is a messy composite of gravity, inertia, the robot's own motion, and the actual contact force. Historically, it has been unclear if we can build a reusable representation of these signals that generalizes across different objects, grasp configurations, and environments.
Methods
The authors propose Language-Grounded F/T Pretraining. The core idea is to treat F/T histories as a "modality" that can be aligned with natural language, similar to how CLIP aligns images with text.
- Automated Simulation: They used the Genesis physics engine to generate over 1.5 million windows of F/T data from 40,000+ pick-and-place episodes. Because it's a simulation, they have "privileged" access to the ground truth (e.g., "the object is slipping," "the surface is wobbling").
- Contrastive Learning: They trained an F/T encoder (evaluating PatchTST, CNN, and MLP architectures) to map 0.8-second signal histories into a shared embedding space with a frozen Sentence-BERT text encoder.
- Zero-Shot Readout: At inference time, the robot can "query" its physical state by comparing the current F/T embedding to text prototypes like "Placing an object" or "Holding a heavy object" using cosine similarity.
- Real-World Transfer: To bridge the sim-to-real gap, they used a simple unlabeled calibration step to normalize real sensor data to the simulated distribution, keeping the pretrained encoder weights frozen.
- Adaptation: For tasks not seen during pretraining (like connector insertion), they used LoRA (Low-Rank Adaptation) to efficiently fine-tune the representation with very few real-world examples.
Datasets
- Pretraining (Sim): 437,244 windows from 40,851 episodes involving 892 Google Scanned Objects (GSO).
- Evaluation (Sim): 111,355 windows involving 75 YCB objects (completely unseen during training).
- Real-World Placement: 80 trials across different objects (wooden cuboid, cork cylinder, compliant ball) and support conditions (fixed vs. wobbling).
- Connector Insertion: A custom real-robot dataset of 125 successful episodes for evaluating few-shot adaptation.
Key Insights and Ideas
- Language as a Semantic Anchor: Aligning F/T signals with language doesn't just make them "queryable"—it provides a structured prior. The authors found that language-prototype readout actually outperformed standard multi-task supervised learning on unseen objects, suggesting that the semantic structure of language helps the model generalize physical concepts.
- Beyond Simple Thresholds: Traditional contact detection often relies on simple rules (e.g., "stop when vertical force reverses"). This fails on compliant objects (like a soft ball) or moving surfaces. The pretrained model, however, learns to recognize the dynamics of load transfer, succeeding where rule-based baselines fail.
- Adaptable Initialization: For entirely new events (like a connector becoming "fully seated"), the pretrained representation is most valuable as a starting point for adaptation (LoRA) rather than a fixed feature extractor.
Key Results
- Generalization: In simulation, the model achieved high Macro F1 scores (>93%) for recognizing actions and object weights on objects it had never seen before.
- Real-World Superiority: In real-world contact detection, the proposed method achieved an 83.3% success rate across various objects, significantly outperforming a vertical-force rule-based baseline (66.7%). Notably, the baseline had a 0% success rate on compliant balls and wobbling supports, while the pretrained model remained functional.
- Data Efficiency: For the new task of connector insertion, using LoRA adaptation on the pretrained model with just one training episode reached ~91.7% F1, whereas training from scratch only reached ~77.5%.
Limitations and Future Work
- Task Diversity: The pretraining was limited to pick-and-place. It doesn't yet cover more complex "contact-rich" maneuvers like sustained sliding, rotation, or manipulating deformable objects.
- Calibration: The system still requires a brief unlabeled calibration phase to handle the specific offsets and scales of real-world sensors.
- Failure Modes: The current evaluation focused on successful episodes; the authors note that future work needs to address failure discrimination and recovery.
- Language Robustness: The study used a finite set of predefined text templates and did not test for robustness against varied phrasing or paraphrasing.
Summary updated