Objectives as Policies: Vision-Language Model Cost Programs for Contact-Rich Manipulation

Zihao Lu, Trong Gia Hung Nguyen, Fuquan Wang, Yifan ZhuarXiv 2026 · arXiv

TL;DR

Objectives as Policies (OaP) bridges the gap between high-level semantic reasoning and low-level physical control by using Vision-Language Models (VLMs) to write editable "objective programs." By iteratively refining these programs in a simulation-based digital twin before real-world deployment, OaP enables robots to perform complex, contact-rich manipulation tasks—like pushing objects with tools or reorienting cups—without any task-specific training data.

What Problem Does This Paper Address?

Robotic manipulation is often a tale of two worlds. On one side, Vision-Language Models (VLMs) excel at semantic understanding (e.g., "put the eraser on the box"), but they struggle with the precise, non-differentiable physics of contact-rich tasks. On the other side, Model-Predictive Control (MPC) handles complex physics beautifully but requires painstakingly hand-engineered reward functions for every new task and environment.

Connecting these two is difficult because there is no common language that captures both the intent of a task and the physics required to execute it. Existing end-to-end "Vision-Language-Action" (VLA) models often fail at contact-rich tasks because they lack an explicit physics model, while previous attempts to use LLMs to script MPC often require frequent, slow updates during execution.

Methods

The authors introduce Objectives as Policies (OaP), a framework that uses a VLM as a "programmer" for a physics-based controller.

  1. Digital Twin via Real-to-Sim: The system first builds a simulation-based replica of the real-world scene using off-the-shelf perception tools (like SAM 3D and FoundationPose).
  2. Objective Programs: Instead of predicting raw actions, the VLM generates a structured JSON program. This program decomposes a task into a sequence of stages. Each stage defines:
  3. Running Costs: Objectives that guide the robot's motion (e.g., "move toward the cup").
  4. Terminal Costs: Conditions that must be met to finish the stage (e.g., "the cup is tilted 20 degrees").
  5. Cost Library: To keep the search space manageable, the VLM selects from a predefined library of parameterized functions (distance, alignment, contact indicators).
  6. Iterative Refinement: This is the "secret sauce." The VLM doesn't just guess once. It performs a self-review of its draft, then executes the program in the digital twin. If the simulation fails, the VLM analyzes the rollout statistics (like residual errors and contact states) to update and fix the program.
  7. MPC Execution: A sampling-based MPC (specifically MPPI) executes the final program. Because it runs in a GPU-parallel simulator (MuJoCo-warp), it can evaluate thousands of potential trajectories in real-time to find the optimal path.
The OaP framework interfaces semantic understanding with physics-based planning through a structured objective program.

Datasets

The framework was evaluated on four tabletop manipulation tasks designed to test contact-rich interactions:

  • Pick-and-place: Moving an eraser onto a box.
  • Cup reorientation: Grasping and tilting a cup.
  • Direct push: Pushing a box to a target.
  • Push with Tool: Using an eraser to push a box.

The authors compared OaP against several strong baselines, including Language to Rewards (L2R), ReKep, CoRAL, and the VLA model OpenVLA.

Key Insights and Ideas

  • The Program as an Interface: By forcing the VLM to output a structured program rather than raw rewards or actions, the authors create a human-readable and editable bridge. This allows for "debugging" the robot's behavior in simulation.
  • Refinement is Crucial: The paper shows that "zero-shot" generation is often insufficient. The ability of the VLM to "see" why a plan failed in simulation and adjust the cost weights or add a "settling" stage is what drives high success rates.
  • Physics Matters: Unlike VLA models that try to learn everything from pixels, OaP leverages the fact that we already have excellent physics simulators. The VLM just needs to tell the simulator what to care about.
The method overview shows the loop of real-to-sim, VLM program generation, and iterative refinement.

Key Results

  • Simulation Success: In simulation, OaP achieved a 90% success rate with just two rounds of feedback, rising to 97.5% with five rounds.
  • Real-World Robustness: When deployed on a Flexiv Rizon 4s arm, the system maintained an 80% success rate across all tasks, even when object locations were perturbed.
  • Baseline Comparison: OaP significantly outperformed VLAs (which achieved 0-25% success) and outperformed CoRAL (77.5%), which requires much more frequent and computationally expensive LLM updates during execution.
Real-world execution examples showing the robot successfully completing multi-stage tasks.

Limitations and Future Work

  • Library Dependency: The VLM is limited by the predefined cost library. If a task requires a physical concept not in the library, the VLM cannot express it.
  • Linear Planning: The current objective programs follow a linear, forward-only sequence. If a stage fails halfway through real-world execution, the controller doesn't have an explicit mechanism to "backtrack" to a previous stage.
  • Sim-to-Real Gap: While the digital twin is effective, errors in estimating friction or object stiffness (e.g., a 3D-printed cup vs. a real one) can still cause failures.
  • Future Directions: The authors aim to incorporate online real-to-sim updates and develop more agentic frameworks that allow for explicit stage recovery during execution.

Summary updated

Abstract

Vision-language models (VLMs) have shown strong semantic and spatial reasoning capabilities in 3D environments. Recently, connecting VLMs with model-predictive controllers (MPC) has emerged as a paradigm for combining strong semantic reasoning with physics-based planning. However, connecting these two for contact-rich robotic manipulation requires a task representation that bridges the two and captures how physical interactions can be reasoned by VLMs. In this work, we introduce Objectives as Policies (OaP), a framework that connects a pretrained VLM to a shared sampling-based MPC through an editable objective program. Each program specifies a sequence of stages for completing a task, combining running and terminal costs from a cost library with stage completion conditions. Given a natural-language task instruction and scene observation, we build a simulation-based digital twin through real-to-sim, and the VLM generates an objective program that is executed by the MPC in the simulation. The program is iteratively updated by the VLM based on MPC rollouts, before being deployed in the real world. We evaluate our framework in both simulation and the real world on four tabletop manipulation tasks, and show that it outperforms strong baselines. Additional videos and examples are available on the project website: https://zihaolu001.github.io/oap-project/
BibTeX
@misc{lu2026objectives,
  author = {Zihao Lu and Trong Gia Hung Nguyen and Fuquan Wang and Yifan Zhu},
  title = {{Objectives as Policies: Vision-Language Model Cost Programs for Contact-Rich Manipulation}},
  year = {2026},
  eprint = {2611.18298},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18298},
}
Figure