Language models can write short programs that control a robot arm: pick up the cup by its handle, open the drawer by pulling along its axis. When such a program fails, the usual fix is to run it in a physics simulator, show the model the error, and ask it to repair the code. These repair loops stall after two or three rounds. Our view is that the stall has a specific cause. A failed physical rollout reports that the mug slipped, not which assumption was wrong, such as the grasp height, the approach direction, or the order of steps. So the model is left guessing. We propose that the model's well-documented sensitivity to harmless rewordings of the instruction can tell us where to look. If five paraphrases of "pour the water into the bowl" produce programs that agree on everything except the tilt angle and the approach height, those two quantities are where the model's physical knowledge is uncertain. They are the place to spend simulator time.
Concretely, we generate policy code from several meaning-preserving paraphrases of the instruction and align the resulting programs call by call. Wherever they agree, we freeze the code. Each point of disagreement becomes an explicit free parameter, and we resolve it by a small simulator search over the candidate values the paraphrases proposed, rather than by another round of free-form language model repair. The plain summary is this: brittleness, usually treated as a defect to suppress, becomes a cheap and training-free locator of uncertain physical beliefs. That locator turns open-ended debugging into a low-dimensional search.
Core Claim
We make two linked claims, and both can be falsified. The first is diagnostic. After a standard repair loop has plateaued, the remaining failures are concentrated in tasks where paraphrase-generated programs disagree on physical quantities (poses, offsets, angles, discrete choices such as grasp axis or object order), not on program structure or API usage. In other words, execution feedback washes out syntactic and API brittleness but leaves physical brittleness intact. The second claim is interventional. Lifting only the disagreeing quantities into a simulator search should raise task success, at a matched simulator budget, above both conversational repair and plain best-of-K selection over the same paraphrase programs. It should also beat lifting every numeric literal in the program.
The second comparison is the important one. If simply running all paraphrase programs and keeping the one that succeeds works as well, then disagreement adds nothing beyond prompt diversity, and we will report it as such.
Grounding in Prior Work
Several lines of work motivate the proposal.
Evidence that models carry usable physical knowledge but that execution is fragile. - Code as Policies has a code model write robot programs over perception and control functions. It is one-shot, and its authors note heavy dependence on engineered prompts. - PromptBook shows that prompt format alone moves real drawer-opening success between 50% and 71%. It also shows that wording changes the inferred grasp geometry. This is direct evidence that brittleness lives in physical parameters, although that paper never measures it systematically or closes the loop. - CoPa shows that vision-language models know which object part matters. They are weaker on metric 3D quantities, which is exactly where we expect paraphrase disagreement to cluster.
The repair-loop tradition, and its documented plateau. - RobotGPT repairs PyBullet manipulation code with a corrector agent for up to three rounds. It treats the prompt as fixed while acknowledging that small prompt changes give inconsistent outputs. - CodeTransOcean reports compiler-feedback debugging rising only from 48.57% to 52.57% over three rounds. - NESYRO adds formal verification and exploratory probes. That requires hand-built symbolic domain models, which we avoid.
We reuse RobotGPT's simulator loop as a component and as a baseline. We differ from all three in what the loop is allowed to change: a small set of located quantities rather than the whole program.
The method supply. - ProSA shows that sensitivity tracks low decoding confidence and peaks on code generation. It also shows that equivalent prompts fail on disjoint instance subsets. We take its paraphrase categories to build our perturbations. - On the Relation between Sensitivity and Accuracy in In-Context Learning uses disagreement across paraphrases to decide when a classifier should abstain. We extend that per-query signal to a per-location signal inside a generated program, and we use it to direct search rather than to abstain. - Selective Prompt Anchoring argues that models lose track of the instruction as generated text grows, and counters this by amplifying the instruction's effect on the next-token scores. It offers a competing explanation of the plateau, which we test head on.
Prior work that constrains our design. RobotGPT notes that its LLM-written success checks needed human oversight. Separately, When Benchmarks are Targets shows that LLM judgments shift under formatting changes. For both reasons we use only simulator ground-truth success predicates, never LLM-written evaluators.
Research Program
The system has three parts.
1. Paraphrase sets. For each task instruction we build a set of $K$ (default 8) meaning-preserving paraphrases. These cover lexical substitution, reordering of clauses, and ProSA-style template changes (plain request, role framing, output-format requirement). Each set is human-checked for equivalence of the task.
2. Program alignment. All programs are written against a fixed robot interface in the Code-as-Policies style. This keeps them comparable at the level of function calls. We parse each program, align the call sequences, and mark two kinds of disagreement:
- Value disagreements: the same call with different numeric or categorical arguments.
- Structural disagreements: different call sequences. These are retained as a small set of discrete alternatives.
3. Repair by located search. Each value disagreement becomes a slot whose search range is spanned by the paraphrases' proposals, padded by a fixed margin. We run a budgeted search over the slots in the simulator: grid search for one or two slots, a cross-entropy method otherwise. Everything else stays frozen. If the search fails, the simulator outcome is handed back to the model for one conventional repair turn, and the cycle repeats.
The analysis side runs alongside the system. At every turn of a standard repair loop, we measure residual sensitivity, defined as the fraction of tasks whose pass or fail outcome still flips across paraphrases. We also classify where the paraphrase programs disagree at that turn: syntax, API usage, structure, or physical value. This directly answers the open question of whether feedback stabilizes or amplifies brittleness across turns.
To check that disagreement actually locates the cause of failure, we run an oracle intervention. For failed programs, we set a single disagreeing slot to a value known to work and see whether success follows. We compare this against setting a randomly chosen agreeing literal. The localization rate is reported for both.
Baselines
Reference points. - Naive reference: one-shot Code-as-Policies generation from the original instruction, executed once. - Prompting-quality control: the same, but with the best of PromptBook's three prompting styles (examples, instructions, chain of thought) selected on a held-out split. This ensures we are not beating a carelessly prompted model.
The strongest existing method: a RobotGPT-style corrector loop. We run it on the same tasks with simulator tracebacks and failure states fed back to the model. The cap is raised from three rounds to whatever round count consumes our full simulator budget. For reference, RobotGPT reports direct ChatGPT execution at 38.5% success in its BulletArm setting, falling to 8% on complex tasks. Its 91.5% figure is for a student policy trained on the corrected demonstrations, not for repaired code itself. We found no reported result on Lacuna for code-level success of the corrector loop alone, so we will measure it ourselves. On RLBench, NESYRO reports a 46.2% success-rate improvement over Code as Policies. We cite this as context rather than rerunning it, because it requires domain models we do not build.
The competitor that most threatens the idea: best-of-K selection. We run two variants at an identical rollout budget:
- programs from $K$ paraphrases, each executed, keeping any that succeeds;
- $K$ temperature samples from the original prompt, handled the same way.
Anchored repair. On an open-weight code model (DeepSeek-Coder-6.7B, matching the Selective Prompt Anchoring setup), we add the corrector loop with the instruction anchored at every repair turn. This uses the anchoring strength tuned on a held-out task split with the same tuning budget we give our own search hyperparameters.
Ablations of our key ingredient. - Lift-all: every numeric literal becomes a slot, searched with the same budget. - Lift-random: a randomly chosen set of slots, matched in size to the disagreement set. - Shuffled ranges: the disagreement slots are kept, but their ranges come from another task.
What counts as beating the competition. All methods share the same tasks, seeds, model, and number of simulator rollouts. Beating the baselines means higher success at a matched rollout budget, with paired bootstrap confidence intervals over tasks that exclude zero, against both best-of-K variants and lift-all.
Why not just…?
Why not just sample K programs and keep whichever succeeds in simulation? This is the real threat, because paraphrase disagreement is partly a diversity mechanism. Our bet is that best-of-K wastes budget: it re-tests entire programs that differ in many places, while the correct value often lies between or near the proposals rather than in any one of them. Metric 3D quantities are exactly the weakness CoPa reports. The matched-budget comparison against paraphrase best-of-K settles this directly. If our method fails to beat it, the contribution shrinks to the diagnostic study, and we will say so.
Why not just anchor the instruction during repair, as Selective Prompt Anchoring does? Anchoring assumes the plateau comes from the model forgetting the instruction. Our hypothesis is that the model remembers the instruction but is uncertain about physical quantities the instruction never states. These explanations predict different things. If dilution is the cause, anchored repair should rescue failures regardless of the type of disagreement. If our explanation holds, anchoring should help mainly on structural and API failures and leave value-disagreement failures stuck. The turn-by-turn breakdown separates the two. Combining anchoring with located search is a natural follow-up if both help.
Why not just engineer a better prompt or add examples? ProSA finds that one-shot examples stabilize models but leave capable models volatile on code. The sensitivity–accuracy paper shows that reordering the examples alone still produces informative disagreement. Our best-of-three PromptBook control measures how much careful prompting recovers on these tasks.
Why not just expose all parameters and optimize them in simulation? For short programs this might work. The lift-all ablation tests it at equal budget. We expect it to degrade as programs grow, because the search dimension grows with every literal while the disagreement set stays small. We will report success as a function of program length to show where, if anywhere, the two diverge.
Proposed Experiments
Environments and tasks. - Primary: the BulletArm PyBullet benchmark (BulletArm), using the open-loop pick-and-place interface that RobotGPT used, plus its harder geometric tasks such as bin packing and house building. - Second domain: RLBench, chosen for its long-horizon and physical-interaction categories. - Third set, optional: a subset of VIMA-BENCH task templates (VIMA), re-expressed as language instructions over a code interface. We use these only to test whether findings transfer to procedurally varied object placements.
Models. One closed and one open-weight code model.
Metrics. - Ground-truth task success as a function of simulator rollouts. - Residual sensitivity per repair turn. - The share of residual failures carrying value versus structural disagreement. - The oracle localization rate.
Decisive checks. There are three, and each corresponds to a falsifiable claim:
- After the corrector loop plateaus, value-disagreement tasks should make up a clear majority of remaining failures. They should also show a much higher oracle localization rate for disagreeing slots than for random agreeing literals.
- Located search should beat paraphrase best-of-K at matched budget.
- Located search should beat lift-all and lift-random, which confirms that the location information, not search alone, does the work.
Failing the first check refutes the diagnostic claim. Failing the second or third reduces the method to prompt-diverse sampling. A small real-arm pilot on drawer opening, the PromptBook task with the widest reported spread, would follow only if the simulator results hold.
References
- Code as Policies: Language Model Programs for Embodied Control
- How to Prompt Your Robot: A PromptBook for Manipulation Skills with Code as Policies
- CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Models
- RobotGPT: Robot Manipulation Learning From ChatGPT
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- Towards Reliable Code-as-Policies: A Neuro-Symbolic Framework for Embodied Task Planning
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- Selective Prompt Anchoring for Code Generation
- When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards
- BulletArm: An Open-Source Robotic Manipulation Benchmark and Learning Framework
- VIMA: Robot Manipulation with Multimodal Prompts