Distractor-Anchored Errors in LLM-Written Planning Domains

Robot planners that reason symbolically need a hand-written "domain model": a formal list of the facts that can hold in the world and the actions that change them, usually written in the Planning Domain Definition Language (PDDL). Recent systems have large language models (LLMs) write these models from text or demonstrations, and then repair them in a loop using feedback from a plan checker. All of these systems are tested on clean, curated task descriptions.

Two other findings suggest this is a weak spot. LLMs are easily distracted by irrelevant text. They are also pulled toward errors that appear earlier in their own context. We propose to test what happens when the description of a planning domain contains realistic clutter, meaning irrelevant objects, incidental properties and misleading hints. We expect the LLM to invent spurious facts and preconditions tied to that clutter. We further expect the usual repair loop to keep those errors alive, because each round re-reads the flawed draft. We will measure both effects against ground-truth domains and test a repair procedure that regenerates only the faulty operators without showing their flawed text.

Core Claim

Distraction and hallucination can be measured exactly in PDDL generation, which math or question-answering benchmarks cannot offer. Every predicate (a named fact such as "the block is on the table") and every precondition (a fact that must hold before an action can run) in a generated domain can be checked against a ground-truth domain. That check shows whether the item corresponds to something real or was anchored on distractor text.

We make two falsifiable claims.

  1. Distractors produce a specific kind of error. Clutter causes over-constrained or spurious symbols ("constraint hallucination"), not just random syntax or semantic errors. The rate of these errors should rise with distractor density, following the power-law trend that How Is LLM Reasoning Distracted by Irrelevant Context? reports for math word problems.
  2. Standard repair keeps those errors. Feedback-driven repair that places the full prior draft in context preserves distractor-anchored symbols across rounds. This mirrors the "contextual drag" effect that Contextual Drag documents: outputs stay structurally close to erroneous drafts, and iterative refinement can collapse.

The proposed fix. We call it quarantined repair. When a checker flags a failure, only the flagged operators are regenerated. The model sees the original description, the checker's message, and the signatures of the remaining operators, but not the flawed text of the operators it is fixing. We claim this beats both repair with the full draft and restarting from scratch at equal compute.

Research Program

The benchmark. Its backbone is the classic, obscure and novel STRIPS suite (blocks, miconic, hiking, pacman-63, pacman-72, flow) from Model Space Reasoning as Search in Feedback Space for Planning Domain Generation. That paper generates domains from natural-language descriptions and refines them with tree search over feedback from the VAL plan validator and from action landmarks (actions every valid plan must contain). It scores results with Heuristic Domain Equivalence (HDE), and it uses only clean descriptions.

Four families of distractors. Following the controlled design of the GSM-DC benchmark, we inject distractors at densities from 1 to 15:

  • irrelevant entities, such as a lamp in a blocks world;
  • plausible but causally inert properties, such as a colour or weight that no action depends on;
  • narrative clutter, the kind Less is More found misleads program synthesis;
  • erroneous hints, such as "remember a block can only be picked up if the table is dry."

Because each injected span is known, every generated predicate and precondition can be attributed to either real description text or distractor text.

What we measure.

  • HDE and plan success on generated problems.
  • A distractor-anchoring rate: the fraction of generated symbols with no ground-truth counterpart whose lexical and semantic match falls on an injected span.
  • Persistence across repair rounds: whether an anchored symbol present at round k survives to round k+1.
  • Tree edit distance between each repaired domain and the round-zero draft, adapted from Contextual Drag.

The interventions. We will build quarantined repair and a matched variant that shows the flawed operator text. This pair is the ablation that isolates the drag mechanism.

Oracle dependence. Model Space Reasoning's VAL feedback compares against a ground-truth domain. We will therefore also run an oracle-free setting, in which feedback comes only from whether a planner can reach goals on sampled problem instances. This checks that the conclusions do not rest on an oracle.

Models. We will use gpt-5-mini, matching Model Space Reasoning, plus a small open model and a reasoning model. This tests whether scale removes the effect.

Baselines

The naive reference: single-shot generation with no feedback. For flow with gpt-5-mini, Model Space Reasoning reports a no-feedback HDE of 33±8% on clean descriptions. That figure anchors how much noise lowers the starting point.

The strongest existing method: Model Space Reasoning's Landmark + Plan Feedback with Search (LVS). The paper reports that LVS achieved the highest average HDE and was the only method to reach a perfect 100% HDE domain on every tested domain at least once. No exact per-domain averages for LVS were found on Lacuna, so we will rerun it ourselves on both clean and noisy descriptions. It is the strongest competitor because it already combines symbolic feedback with search over repair sequences. Its drafts are conditioned on prior flawed domains, which is exactly the mechanism we suspect.

For fairness, every method gets the same descriptions, the same LLM, the same feedback source and the same cap on LLM calls and tokens per domain. Temperature and prompt wording receive the same small tuning budget for all methods on held-out domains. Quarantined repair beats LVS if, at equal calls, it reaches higher mean HDE at medium and high distractor density, with non-overlapping confidence intervals across seeds, while matching LVS on clean descriptions.

Prompt-level defences.

Restart. Best-of-N fresh generation, filtered by the same checker at equal call budget.

Key ablation. Quarantined repair with the flawed operator text shown versus withheld. This separates the effect of hiding the draft from the effect of localizing the repair.

Why not just…?

Why not just summarize or filter the description before generating? This is a real threat for narrative clutter, and we expect it to help there. The difficulty is inert properties and erroneous hints that are phrased like genuine constraints. A summarizer cannot tell "blocks must be dry" from "blocks must be clear" without knowing the domain. Contextual Drag also found that its REVISE/FILTER denoising only partly restored accuracy. To test the other side of the trade-off, we include a control set where the "distracting-looking" detail is actually required. If summarization matches quarantined repair on both the distractor and control sets, we will report that the fix belongs upstream, not in repair.

Why not just restart from scratch instead of repairing? Restarting avoids drag completely, so it is the most direct competitor. Its cost is that it throws away the operators that were already correct. We predict restart wins on small domains and loses on large ones, where getting every operator right at once is unlikely. Running both at equal call budgets, broken down by domain size, will settle this. If restart matches quarantined repair everywhere, the finding simplifies to "never repair in context," which is still useful.

Why not just use a stronger reasoning model? Contextual Drag reports that even frontier models show non-negligible degradation. One Demo Is All It Takes shows reasoning models spend three to four times more tokens on planning. We will still include a reasoning model. If it shows near-zero anchoring at high density, the effect is a small-model issue and our claim narrows accordingly.

Why not prune irrelevant entities as Flax does? Fast Task Planning with Neuro-Symbolic Relaxation prunes objects at planning time using hand-crafted relaxation rules. It acts after the domain exists and cannot remove a spurious precondition already written into an operator.

Proposed Experiments

Experiment one: the dose-response curve. For each model and distractor family, we plot HDE and anchoring rate against density on single-shot generation. Claim one is supported if anchoring rises with density and accounts for most of the HDE loss. It is rejected if errors are unrelated to the injected spans.

Experiment two: the persistence test. We run LVS and plain full-draft repair for five rounds, tracking survival of anchored symbols and edit distance to the round-zero draft. Drag is confirmed if anchored symbols survive at higher rates than other errors of similar checker visibility, and if edit distance stays low even as feedback accumulates. Any cases where HDE declines across rounds would count as self-deterioration.

Experiment three: the decisive comparison. Quarantined repair against LVS, restart, summarize-then-generate and the shown-text ablation, at equal budget, under both oracle and oracle-free feedback.

  • The central prediction: withholding the flawed text beats showing it at identical localization.
  • If withheld and shown perform the same, contextual drag is not the operative mechanism in formal domain repair, and the gain, if any, comes from localization alone.
  • The proposal fails outright if withholding the text does not help and localization gains disappear against restart.

A transfer check reruns the best configuration on the unseen Self-CriTeach task descriptions (Prepare Experiment, Reorganize Room, Machine Parts Assembly) with injected clutter. This tests whether the effects carry over from benchmark STRIPS domains to robotic task specifications. It uses plan success, since ground-truth domains for HDE may be unavailable there.