TL;DR
TacBridge is a new framework that creates a "universal language" for diverse optical tactile sensors by training on paired videos of the same contact events. By learning to translate interactions between different sensor types (like GelSight and DIGIT), it achieves state-of-the-art performance in robotic perception tasks and enables high-fidelity cross-sensor video generation.
What Problem Does This Paper Address?
In the world of robotic touch, we have a "Tower of Babel" problem. Different optical tactile sensors—such as GelSight, DIGIT, and GelSlim—all record contact differently. They vary in illumination, elastomer material, and camera geometry. Consequently, a robot trained to recognize a silk fabric using a DIGIT sensor might be completely lost if you swap it for a GelSight Mini.
This fragmentation makes it incredibly difficult to combine datasets or transfer learned skills across different hardware. While previous research attempted to align these sensors by looking at broad data distributions, TacBridge argues that we need a more direct approach: event-level correspondence. If we show a model exactly how the same physical interaction looks through two different sensors, it can learn to separate the "content" of the touch from the "style" of the device.
Methods
The authors introduce a three-stage pipeline to bridge the gap between sensors:
- Multi-Sensor Pre-training: They first train a Spatiotemporal Masked Autoencoder (MAE) on a massive corpus of tactile videos from various sensors. The model learns to "fill in the blanks" of masked video patches, forcing it to understand both spatial contact patterns and how they evolve over time.
- Event-Paired Learning: This is the core innovation. Using datasets where the same contact event is recorded by multiple sensors (or simulated across multiple sensor profiles), the model is trained to reconstruct a "target" sensor's view using only the "source" sensor's data. This teaches the encoder to extract sensor-invariant physical features.
- Controllable Tactile Generation: To prove the representation is truly unified, they build a TacDiT (Tactile Diffusion Transformer). This latent flow model can take a video from one sensor and "render" it as if it were seen by another, conditioned on the target sensor's identity and background.
Datasets
The study leverages a mix of real-world and simulated data:
- TacQuad: A real-world dataset providing synchronized observations from DIGIT, GelSight Mini, and DuraGel sensors.
- ToucHD-Sim: A simulation environment that renders the same interaction under nine different optical sensor appearances.
- Pre-training Corpus: A massive collection of public datasets including Touch and Go, VisGel, and ObjectFolder Real, covering a wide array of objects and sensors.
- Real-Robot Deployment: A custom dataset of a robot arm pressing 50 different fabrics to test real-world generalization.
Key Insights and Ideas
- Correspondence is King: The authors show that explicitly linking paired observations is far more effective than just training on a mix of unpaired data. It provides a "ground truth" for what the underlying physics should look like regardless of the camera.
- Temporal Dynamics Matter: By using tactile videos rather than static images, TacBridge captures the "evolution" of a touch (establishment, maintenance, and release). This proved critical for dynamic tasks like slip detection.
- Latent Translation: The ability to translate features in a shared latent space allows for "zero-shot" style transfer between sensors. You can effectively "see" through a sensor you don't even own by translating your current data into its domain.
Key Results
TacBridge outperformed strong baselines (like AnyTouch and Sparsh) on 9 out of 11 downstream metrics.
- Dynamic Perception: The most dramatic gains were in dynamic tasks. For example, on the DIGIT sensor, slip detection F1-scores jumped from 40.60 (AnyTouch) to 85.77 (TacBridge).
- Force Estimation: RMSE for force prediction was reduced by 17% to 69% compared to the strongest baselines.
- Real-World Generalization: When tested on a real robot for fabric attribute classification (e.g., is it "fuzzy" or "stiff"?), the model achieved 73% accuracy on entirely unseen fabrics, demonstrating that the learned features generalize well beyond the training set.
Limitations and Future Work
While TacBridge is a significant step toward a "Foundation Model" for touch, the authors note several areas for growth:
- Sensor Variety: The current evaluation covers a limited range of sensors. Expanding to more diverse hardware (like capacitive skins) is a logical next step.
- Real-World Fidelity: While the cross-sensor translation works well in simulation, achieving the same level of photorealism on real-to-real sensor pairs remains a challenge.
- Closed-Loop Control: The current work focuses on "offline" perception. The ultimate goal is to integrate these unified representations into real-time, closed-loop robotic manipulation.
For more details, you can find the full paper "TacBridge: Learning Unified Tactile Representations through Cross-Sensor Correspondence" on arXiv.
Summary updated