While frontier multimodal models like GPT-4o or the hypothetical "GPT-6 Astra" mentioned in this paper are surprisingly good at controlling robots using just images, they are effectively "numb." They can see a screw, but they can't feel if it has caught the thread. They can see a bottle, but they can't tell if it’s full or empty by its weight.
ForceHarness bridges this gap, allowing "frozen" (unmodified, API-only) models to master contact-rich manipulation tasks without any task-specific training or fine-tuning.
TL;DR
ForceHarness is a modular system that enables frontier AI models to perform complex robotic tasks by handling high-speed force control locally and translating raw sensor data into meaningful physical quantities. By "closing the loop" beneath the model, it raises the zero-shot success rate of a frontier model on real-world contact tasks from a measly 3.3% to a robust 90.0%.
What Problem Does This Paper Address?
Current Large Multimodal Models (LMMs) face two major hurdles when it comes to haptic feedback:
- The Latency Gap: Contact forces change in milliseconds. If a robot hits a surface, it needs to react instantly to avoid damage. API-based models, which take seconds to process a single frame, are far too slow to be in the direct control loop for force.
- The Interpretation Gap: A frozen model doesn't know what to do with a stream of raw numbers like
[1.2N, -0.5N, 9.8N]. It needs semantic physical concepts—like "mass," "friction," or "impact"—to make high-level decisions.
Previously, researchers tried to solve this by training specialized "force-aware" policies on thousands of demonstrations. ForceHarness takes a different path: it builds a "harness" around a general-purpose model so it can use force zero-shot.
Methods
The authors propose a three-layer architecture that sits between the AI model and the robot hardware:
- Force Sensing & Calibration: This layer estimates the force at the robot's hand, either from a dedicated wrist sensor or by calculating it from joint torques. It includes a "tare" function to zero out gravity and sensor drift before each action.
- Self-Terminating Force Primitives: Instead of the model controlling every tiny movement, it calls high-level "tools" like
search,push, orscrew. These primitives run locally at high frequency (e.g., 20Hz to 1kHz). They are "self-terminating," meaning they execute until a specific physical event occurs (like "contact felt" or "plug dropped into hole") and then return control to the model. - Physics Translation Layer: This is the "translator." It takes the messy force data from a primitive's execution and distills it into physical quantities with uncertainty estimates (e.g., "Mass: 0.6kg ± 0.03kg").
Datasets
The researchers evaluated ForceHarness across three distinct environments:
- Real-Robot Tasks: Six challenging real-world tasks including driving a screw, peeling an eggplant, and sorting bottles by weight.
- ForceBench: A public benchmark of eight contact-rich tasks in the Isaac Lab simulator.
- HiddenPhys (New): A newly introduced benchmark of seven MuJoCo tasks designed specifically to test haptic reasoning. It "hides" physical variables (like the exact mass of a cube or the height of a table) so the robot must use force to succeed.
Key Insights and Ideas
- Intelligence vs. Reflex: The core philosophy is to separate high-level reasoning (the model) from low-level reflexes (the harness). The model decides how hard to press, but the harness ensures the robot stops pressing the moment it hits something.
- Semantic Physics is Key: Ablation studies showed that smaller models struggle to interpret raw force numbers but perform significantly better when given translated physical quantities like "friction" or "stiffness."
- Zero-Shot > Trained: Surprisingly, a general-purpose model equipped with ForceHarness outperformed specialized policies that were trained on 200 demonstrations per task. This suggests that "understanding" physics is often more powerful than "mimicking" it.
Key Results
The quantitative improvements are dramatic across the board:
- Real Robot: On a Franka FR3 arm (which lacks a wrist force sensor), ForceHarness boosted the success rate of GPT-6 Astra from 3.3% to 90.0%.
- HiddenPhys: In simulation, it raised success from 62.1% to 96.4%, matching the performance of hand-written expert controllers.
- ForceBench: It improved success by 17.5 percentage points over the best-trained baselines.
Limitations and Future Work
While ForceHarness is a major step forward, the authors note several limitations:
- Latency: There is still a multi-second delay between "tools" while the model thinks, which prevents highly reactive behaviors like catching a falling object.
- Fixed Menu: The current system relies on a pre-defined menu of 12 force primitives. Future work could involve letting the model "synthesize" its own primitives or code-based control laws.
- Single-Arm Focus: The current evaluation is limited to single-arm tabletop manipulation. Extending this to bimanual or mobile manipulation remains an open challenge.
For more details, check out the project page at showlab.github.io/ForceHarness.
Summary updated