Reins: Frontier Multimodal Models Driving Real Robot Arms on Long-Horizon Tasks, Zero-Shot

Pengfei Ye, Zhiyang Dou, Yuxiang Ma, Haonan Chen, Heng Yang, Edward H. AdelsonarXiv 2026 · arXiv

TL;DR

Reins is a new "harness" that allows off-the-shelf frontier multimodal models (like GPT-6.1 and Claude 3.5) to drive real-world robot arms zero-shot, achieving an 88% success rate on complex, long-horizon tasks. It overcomes the "blindness" of models during computation by using a dedicated "narrator" instance to track scene changes and "guarded chains" of actions that stop automatically when physical conditions are met.

What Problem Does This Paper Address?

While frontier models have become excellent at high-level planning in simulation, they often fail when translated to real-world robotics. Two major hurdles exist:

  1. The Thinking Gap: These models are slow. While a model "thinks" for 10–30 seconds, the world changes, or the robot moves blindly.
  2. Contact-Rich Execution: Real-world tasks like opening a drawer or folding a shirt require reacting to physical contact and resistance, which is difficult to script in a single open-loop command.

Most existing approaches either require massive amounts of task-specific training data or move the robot in tiny, inefficient steps. Reins aims to bridge this gap by providing a framework that lets a general-purpose model "feel" its way through a task using atomic actions and continuous visual monitoring.

Methods

The Reins framework consists of three primary components:

  • The Actor Loop & Guarded Chains: Instead of sending one command at a time, the "Actor" model outputs a chain of actions (e.g., "move down until you hit something, then close the gripper"). These are "guarded" by conditions—if a condition fails (e.g., the gripper didn't actually catch anything), the chain stops immediately and reports back. This mimics classical "guarded moves" in robotics but is generated dynamically by the LLM.
  • The Narrator: Because the Actor is "blind" while it processes a frame, a second instance of the model—the Narrator—watches the camera feed at 1Hz. It keeps a running text account of what changed (e.g., "the red mug moved to the left"). This account is fed to the Actor with the next result, ensuring it stays updated on the state of the world.
  • Digital Twins: Reins can automatically build a simulation-ready MuJoCo environment from a single RGB head-camera frame. It uses Segment Anything (SAM 3) to identify objects and fits them to parametric 3D models, allowing for "continual learning" where successful real-world episodes can be rehearsed and refined in simulation.
Figure 1: An overview of Reins. Left: three calls of a Sol drawer episode; results report what the robot measured, a chain of pulls stops when its condition fails, and the narrator’s account arrives with the next result. Middle: the five tasks. Right: sim-ready digital twins, each built from one head-camera frame, at rest and running in MuJoCo (App. C), for continual learning; only object dynamics are shown.

Datasets

The authors evaluated Reins on five challenging, long-horizon real-world tasks using two 6-DoF AirBot Play arms:

  1. Shell Game: Tracking a hidden object under three mugs during a shuffle.
  2. Drawer: Opening a drawer by a tiny lip, placing a block inside, and closing it.
  3. Towel: Folding a towel twice.
  4. Shirt: Folding sleeves in and then folding the shirt twice.
  5. Sweep: Using one arm with a dustpan and the other with a brush to collect objects.

They benchmarked three frontier models: GPT-6.1 Sol, GPT-6 Astra, and Claude Opus 5.5.

Key Insights and Ideas

  • Latency is the Enemy: In their runs, model latency accounted for 41–78% of the total episode time. Without the Narrator to bridge this gap, models frequently lost track of objects in dynamic tasks like the shell game.
  • The Power of "Guarded" Actions: The "harness" is essential. Without it (running raw images and open-loop moves), GPT-6.1 failed every single attempt at the shell game, often closing its gripper on empty air repeatedly.
  • Narrator Tier Matters: Faster models (like Sol on an "Ultrafast" tier) make better narrators because they can process more "windows" of time, seeing shuffles that slower models might miss or report too late.
  • Identity Persistence: One of the most impressive feats was the Narrator's ability to maintain object identity through shuffles (e.g., "the original middle mug") using only natural language descriptions.

Key Results

The quantitative results highlight the effectiveness of the Reins harness:

  • Success Rates: GPT-6.1 Sol achieved an 88% success rate across all tasks.
  • The "Narrator" Effect: In the Shell Game, Sol succeeded 5/5 times with the narrator but only 2/5 times without it.
  • Efficiency: Using the harness allowed Sol to complete the Shell Game in just 1.1 minutes on average. Without the harness, it took 8.6 minutes and never succeeded.
  • Model Comparison: While all three frontier models could handle the static "Drawer" task perfectly, they diverged on the dynamic "Shell Game," with Sol outperforming Astra and Opus.
Success rates, median times, and API costs for the different models and tasks.

Limitations and Future Work

  • Sensing: Currently, contact is only sensed through joint current and gripper width. More sophisticated tactile sensing could improve performance on delicate tasks.
  • Prompt-Based Knowledge: The robot's "understanding" of the task is limited to what is in the natural language prompt.
  • Sample Size: The evaluation used five episodes per condition; larger-scale testing would provide more statistical weight.
  • Future Directions: The authors plan to turn successful real-world episodes into "reusable chains" that can be rehearsed in the digital twins, potentially allowing the robot to finish tasks faster with shorter prompts in the future.

Summary updated

Abstract

Frontier models such as GPT-6.1 Sol, GPT-6 Astra and Claude Opus 5.5 can now plan and call tools, yet they still struggle to drive real robots through long-horizon, contact-rich manipulation, blind while the arm moves and while they think. We present Reins, a harness in which an off-the-shelf model operates a dual-arm robot with 3D perception tools and atomic actions that report contact and grip. The model chains actions in one call, each with a condition that must hold, stopping at the first that fails; a second instance, the narrator, watches the cameras continuously and reports what changed with every result. From images, Reins also builds a simulation-ready twin of the scene for continual policy learning. With natural-language prompts and no task-specific code, GPT-6.1 Sol succeeds in 88% of real episodes on five tasks; without the harness, it fails every shell game, taking 8 times as long. Three frontier models perform alike but differ in tracking a scene that changes while the actor thinks. Code will be released upon publication.
BibTeX
@misc{ye2026reins,
  author = {Pengfei Ye and Zhiyang Dou and Yuxiang Ma and Haonan Chen and Heng Yang and Edward H. Adelson},
  title = {{Reins: Frontier Multimodal Models Driving Real Robot Arms on Long-Horizon Tasks, Zero-Shot}},
  year = {2026},
  eprint = {2611.16632},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16632},
}
Figure