When a person tells a robot "put the red block in the empty spot next to the blue one," the target is a place, not an object. Vision-language models (systems that read images and text together) handle this poorly. On a tabletop block-world benchmark, predicting the target position was much harder for them than picking the source block. The best fine-tuned model reached 54% accuracy on user corrections, against 75% for humans (Repairs in a Block World). Our proposal is to stop asking the model for a coordinate. Instead, it writes a tiny program that builds the target from things that do exist: "the cell one step left of block 7, at table height." Two consequences follow. The program can only mention real blocks, because a grammar enforced during decoding forbids anything else. And the program can be run and checked: is that cell already occupied, is it floating, is it off the table? When a check fails, the system repairs the one faulty clause rather than guessing a new number.
In two sentences, the idea is this. Represent a non-existent target as a landmark-relative program decoded under a scene-specific grammar. Then refine that program in a loop driven by cheap physical-validity checks. We expect that turning "where nothing is" into "relative to what is" makes the target both easier for the model to express and possible to verify. Verifiability is what lets execution-feedback refinement work on a problem that coordinate regression cannot debug.
Core Claim
The claim is that relational anchoring plus localized, executable checks is what closes the gap on empty-target grounding. Neither formatting guarantees alone nor refinement alone should be enough. Three pieces of evidence motivate the anchoring half. Humans grounding arbitrary points in 360° panoramas drop from 88% to 37% accuracy when the intermediate landmark-relative steps are removed (Refer360°). Köhn and Koller generate references to not-yet-built Minecraft structures by anchoring empty coordinates to existing blocks through a synchronous grammar, though with hand-written rules, no neural model, and no feedback (Talking about what is not there). The ontology literature argues that treating location as a property of objects breaks down for empty or coincident regions, and that regions need their own explicit mapping layer (Tucking RCC in Cyc's Ontological Bed; Layered Mereotopology). Our anchor program is a lightweight, executable version of that mapping layer.
The refinement half builds on closed-loop code repair. RobotGPT reports 91.5% simulated manipulation success after simulator-feedback correction, against 38.5% for direct generation (RobotGPT). That work grounds only objects that already exist and uses unconstrained Python. VDebugger shows that execution traces let a critic model point to the faulty line of a visual program (VDebugger), again only for detected, existing objects. Neither system has a representation in which an empty target can fail a check in an informative way. Ours does: a failed check names a specific landmark or offset.
Research Program
The language. We will build a small domain-specific language for targets. A target is a landmark block identifier, a relation drawn from a closed vocabulary (left-of, on-top-of, in-row-with, and so on), and an integer offset. Relations can compose so that a target can be anchored to another target. We include explicit row and column constructs because the benchmark authors report that models fail specifically on "rows" and "columns."
The decoder. Decoding uses an input-dependent grammar, meaning a grammar whose allowed words are rebuilt for each input. The idea comes from Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning, where restricting outputs to entity candidates relevant to the prompt raised entity-disambiguation accuracy from 54.1% to 80.3%. Here, the allowed landmark tokens at each step are exactly the blocks present in the current scene. We will implement this on top of epfl-dlab/GCD. Because the grammar is tiny, we avoid the memory and latency costs reported for full-language systems such as SynCode and Flexible and Efficient Grammar-Constrained Decoding. Our contribution is not the decoder. It is the choice of what the grammar forces the model to say.
The executor and checker. An interpreter resolves the program to a grid cell. It then runs checks that need no ground truth: the cell is unoccupied, the cell is supported, the cell is in bounds, and the source block is not its own anchor. Each failure is reported at the level of the clause that caused it.
The refinement loop. The model receives the failing clause, the checker's message, and, in the benchmark's repair setting, the user's natural-language correction. It edits under the same grammar. The loop is capped at a few rounds, since Yan et al. report that refinement plateaus or oscillates after three to five iterations.
The context question. We will also test a tension the research file surfaces. BW-R finds that training on histories containing wrong intermediate answers hurts, a "negative learning" effect: Idefics2 scored 26% repair accuracy with standard loss, against 47% with completion-only loss. Code-repair systems, by contrast, keep failed attempts in context on purpose. We will compare, at inference and in training, keeping the failed program in the prompt, keeping only the checker message, and masking failures from the loss.
Scene list. The scene list comes from simulator state where the benchmark exposes it. Otherwise it comes from a detector run on the image. We report both settings, so that perception errors are not confused with grounding errors.
Baselines
Naive reference point. The naive reference point is the same vision-language backbone prompted zero-shot to output the target coordinate directly, without a grammar or a loop. This shows how much of the problem is solved by asking at all.
Strongest existing method. The strongest existing method is the fine-tuned LLaVA from Repairs in a Block World, which reports 54% accuracy on repairs. We also include the completion-only-loss Idefics2 at 47%. These are the best published systems on exactly this task, and they are trained end-to-end on it. For fairness, our method uses the same backbone, fine-tuned on the same BW-R training split. Gold anchor programs are derived automatically from the gold target cell by choosing the nearest landmark that describes it unambiguously. Fine-tuning steps and hyperparameter search budget are matched. Beating it means exceeding 54% on the benchmark's repair-accuracy metric with the same backbone, in the detected-scene setting, with the improvement concentrated on target-position errors.
Unconstrained execution-feedback refinement. This is a RobotGPT-style loop (RobotGPT). The model writes free-form placement code or coordinates, the same checker supplies feedback, and the round budget is identical. It isolates whether the grammar and the anchoring matter once feedback is available.
Ablations. Three ablations isolate the key ingredients. The first keeps grammar-constrained decoding but lets the grammar accept raw absolute coordinates, so that format is enforced but anchoring is not. The second prompts for anchor programs without enforcing the grammar. The third removes the checker and lets the model self-critique. The first ablation is the decisive one for our claim.
Why not just…?
Why not just fine-tune the vision-language model to regress coordinates better? That is precisely the strongest baseline, and it stalls at 54% against 75% for humans. The Refer360° results point the same way: an image-to-map model reaches only 13.0% on arbitrary-point grounding. If our anchored version on the same backbone does not beat it, the claim fails. The comparison is built into the main experiment.
Why not just use JSON mode and retry on parse failures? For syntax alone, this is a real threat. A modern model may rarely produce malformed output, and then grammar-constrained decoding adds nothing. That is why our claim does not rest on syntactic validity. It rests on the scene-dependent grammar making it impossible to reference a block that does not exist, and on forcing a relational form. The "prompted anchors, no grammar" ablation settles it. We will report how often unconstrained anchors name non-existent blocks or invalid relations. If that rate is near zero and accuracy matches, we will drop the decoding constraint and say so.
Why not just run a feedback loop on raw coordinates? A checker can say "occupied" about a coordinate. It cannot say which part of the description was wrong, so the model tends to jitter the number. The plateau reported by Yan et al. is the expected failure mode. The unconstrained-refinement baseline and the absolute-coordinate ablation measure this directly. We will track oscillation rate, meaning the fraction of loops that revisit a previous answer, as well as final accuracy.
Proposed Experiments
The primary evaluation is BW-R target-position and repair accuracy, broken down by relation type. Separate breakdowns for row and column instructions test whether the explicit constructs fix the reported weakness. We report accuracy after each refinement round, so the curve, not just the endpoint, is visible.
A secondary synthetic set extends the task to multi-cell targets, such as building a railing along an empty edge in the spirit of Köhn and Koller. This tests compositional anchors. No reported result was found on Lacuna for a neural baseline on such tasks, so the synthetic set will only compare systems within this study.
Refer360° serves as a transfer check for anchoring alone. Landmarks there are detected objects, and the checker is limited to visibility constraints. We compare against the reported 13.0% for LingUNet, with the caveat that the backbones differ.
Decisive checks. Three outcomes would decide the proposal. The anchored, grammar-constrained, checker-refined system must beat 54% with the same backbone. The absolute-coordinate ablation must lose most of that gain. And the context-handling study must show whether keeping failed programs helps or reproduces BW-R's negative-learning effect. If only the first two hold, anchoring is the contribution and refinement is secondary. If only refinement helps, the proposal reduces to RobotGPT on a new domain, and we will report it that way.
References
- Repairs in a Block World: A New Benchmark for Handling User Corrections with Multi-Modal Language Models
- Refer360$^\circ$: A Referring Expression Recognition Dataset in 360$^\circ$ Images
- Talking about what is not there: Generating indefinite referring expressions in Minecraft
- Tucking RCC in Cyc's Ontological Bed
- Layered Mereotopology
- RobotGPT: Robot Manipulation Learning From ChatGPT
- VDebugger: Harnessing Execution Feedback for Debugging Visual Programs
- Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning
- SynCode: LLM Generation with Grammar Augmentation
- Flexible and Efficient Grammar-Constrained Decoding
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation