ForceHarness: Turning Force into Action and Physics for Frozen Frontier Models

Ke Sun, Kevin Yuchen Ma, Yike Zhang, Yanzhe Chen, Mike Zheng ShouarXiv 2026 · arXiv

While frontier multimodal models like GPT-4o or the hypothetical "GPT-6 Astra" mentioned in this paper are surprisingly good at controlling robots using just images, they are effectively "numb." They can see a screw, but they can't feel if it has caught the thread. They can see a bottle, but they can't tell if it’s full or empty by its weight.

ForceHarness bridges this gap, allowing "frozen" (unmodified, API-only) models to master contact-rich manipulation tasks without any task-specific training or fine-tuning.

TL;DR

ForceHarness is a modular system that enables frontier AI models to perform complex robotic tasks by handling high-speed force control locally and translating raw sensor data into meaningful physical quantities. By "closing the loop" beneath the model, it raises the zero-shot success rate of a frontier model on real-world contact tasks from a measly 3.3% to a robust 90.0%.

What Problem Does This Paper Address?

Current Large Multimodal Models (LMMs) face two major hurdles when it comes to haptic feedback:

  1. The Latency Gap: Contact forces change in milliseconds. If a robot hits a surface, it needs to react instantly to avoid damage. API-based models, which take seconds to process a single frame, are far too slow to be in the direct control loop for force.
  2. The Interpretation Gap: A frozen model doesn't know what to do with a stream of raw numbers like [1.2N, -0.5N, 9.8N]. It needs semantic physical concepts—like "mass," "friction," or "impact"—to make high-level decisions.

Previously, researchers tried to solve this by training specialized "force-aware" policies on thousands of demonstrations. ForceHarness takes a different path: it builds a "harness" around a general-purpose model so it can use force zero-shot.

Methods

The authors propose a three-layer architecture that sits between the AI model and the robot hardware:

  1. Force Sensing & Calibration: This layer estimates the force at the robot's hand, either from a dedicated wrist sensor or by calculating it from joint torques. It includes a "tare" function to zero out gravity and sensor drift before each action.
  2. Self-Terminating Force Primitives: Instead of the model controlling every tiny movement, it calls high-level "tools" like search, push, or screw. These primitives run locally at high frequency (e.g., 20Hz to 1kHz). They are "self-terminating," meaning they execute until a specific physical event occurs (like "contact felt" or "plug dropped into hole") and then return control to the model.
  3. Physics Translation Layer: This is the "translator." It takes the messy force data from a primitive's execution and distills it into physical quantities with uncertainty estimates (e.g., "Mass: 0.6kg ± 0.03kg").
Figure 2. Overview of ForceHarness, illustrated with one logged call. Blue marks the command path and teal the force path. The frozen model reads its inputs and returns one tool call, here a search during a USB-plug insertion. The harness runs this call as a self-terminating force primitive that updates the target pose by Eqs. (2) and (3) at every tick and stops on a contact event. The inset sketches how search presses with Fd and follows a spiral until the plug drops into the hole, while the photo and the force trace come from this call. Beneath the primitive, the robot’s stock impedance controller turns the lead into force, and the bottom layer of ForceHarness estimates and re-zeros the hand force with Eq. (1). The same harness drives a real arm and two simulators, and a fixed guard bounds what any primitive can do. Fixed estimators turn the trace into physical quantities with uncertainties and flags, which return to the model as the tool result.

Datasets

The researchers evaluated ForceHarness across three distinct environments:

  • Real-Robot Tasks: Six challenging real-world tasks including driving a screw, peeling an eggplant, and sorting bottles by weight.
  • ForceBench: A public benchmark of eight contact-rich tasks in the Isaac Lab simulator.
  • HiddenPhys (New): A newly introduced benchmark of seven MuJoCo tasks designed specifically to test haptic reasoning. It "hides" physical variables (like the exact mass of a cube or the height of a table) so the robot must use force to succeed.

Key Insights and Ideas

  • Intelligence vs. Reflex: The core philosophy is to separate high-level reasoning (the model) from low-level reflexes (the harness). The model decides how hard to press, but the harness ensures the robot stops pressing the moment it hits something.
  • Semantic Physics is Key: Ablation studies showed that smaller models struggle to interpret raw force numbers but perform significantly better when given translated physical quantities like "friction" or "stiffness."
  • Zero-Shot > Trained: Surprisingly, a general-purpose model equipped with ForceHarness outperformed specialized policies that were trained on 200 demonstrations per task. This suggests that "understanding" physics is often more powerful than "mimicking" it.

Key Results

The quantitative improvements are dramatic across the board:

  • Real Robot: On a Franka FR3 arm (which lacks a wrist force sensor), ForceHarness boosted the success rate of GPT-6 Astra from 3.3% to 90.0%.
  • HiddenPhys: In simulation, it raised success from 62.1% to 96.4%, matching the performance of hand-written expert controllers.
  • ForceBench: It improved success by 17.5 percentage points over the best-trained baselines.
Figure 1. ForceHarness lets a frozen frontier model use wrist force. Left: the model only issues tool calls; the harness calibrates hand force, runs self-terminating force primitives beneath the model, and returns physical quantities. Middle: four real-robot tasks; each column shows one primitive call, the force logged during that call, the quantity its tool result reported, and the model’s next call with a summary of its reason. Right: success rate of GPT-6 Astra with and without force on the real robot, ForceBench, and HiddenPhys, with the best trained policy where one exists.

Limitations and Future Work

While ForceHarness is a major step forward, the authors note several limitations:

  • Latency: There is still a multi-second delay between "tools" while the model thinks, which prevents highly reactive behaviors like catching a falling object.
  • Fixed Menu: The current system relies on a pre-defined menu of 12 force primitives. Future work could involve letting the model "synthesize" its own primitives or code-based control laws.
  • Single-Arm Focus: The current evaluation is limited to single-arm tabletop manipulation. Extending this to bimanual or mobile manipulation remains an open challenge.

For more details, check out the project page at showlab.github.io/ForceHarness.

Summary updated

Abstract

Frontier multimodal models can now control robots zero-shot from images, yet many manipulation tasks depend on contact forces that images cannot show. Bringing force to an API-only frozen model raises two challenges. First, contact evolves much faster than an API model can decide, so the system must close the loop on force beneath the model. Second, a frozen model cannot interpret raw force streams, so force must be returned in a form the model can reason about. Therefore, we present ForceHarness, a harness that lets frozen models use force without demonstrations or finetuning. After calibrating hand force from a wrist sensor or joint torques alone, ForceHarness addresses the first challenge with self-terminating force primitives that regulate force on their own and return at a contact event. To address the second challenge, a physics translation layer turns each force trace into physical quantities with uncertainty as the tool result. To evaluate ForceHarness against expert force controllers and learned policies under identical conditions, we also build HiddenPhys, a seven-task MuJoCo benchmark with hidden physical variables. On a real arm without a wrist force sensor, ForceHarness raises the zero-shot success rate of GPT-6 Astra from 3.3% to 90.0%. In simulation, it adds 17.5 percentage points on the public ForceBench suite and 34.3 percentage points on HiddenPhys, surpassing force-aware policies trained on demonstrations on both benchmarks and matching hand-written expert controllers on HiddenPhys. Ablations further show that smaller models make significantly better decisions when given physical quantities rather than raw force values.
BibTeX
@misc{sun2026forceharness,
  author = {Ke Sun and Kevin Yuchen Ma and Yike Zhang and Yanzhe Chen and Mike Zheng Shou},
  title = {{ForceHarness: Turning Force into Action and Physics for Frozen Frontier Models}},
  year = {2026},
  eprint = {2611.21099},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21099},
}
Figure