Which Repair Works? Factorial Diagnosis of Why Reasoning Models Fail on Unfamiliar Words

Language models that plan well in familiar settings often fall apart when the same problem is described with unfamiliar words. On the planning benchmark PlanBench, OpenAI's o1-preview solves 97.8% of standard Blocksworld problems. When the action names are replaced by arbitrary words such as "feast," it solves 52.8% (LLMs Still Can't Plan; Can LRMs?). The field now measures this drop routinely. What it cannot yet say is what to do about a given failing problem. This is the same trap that predictive models fall into in education and medicine. A risk score tells you that something will fail, not which intervention would prevent it (On the Actionability of Outcome Prediction).

We propose to borrow the method climate scientists use to separate tangled causes, called factorial simulation: re-run the same system many times, switching individual drivers or sub-processes on and off, and compare outcomes. Our proposal applies this to a model's reasoning. For each planning problem we cross the problem's surface factors (which words are obfuscated, whether the instance could have been memorized, how long the plan is) with in-prompt "process switches" (supplying the true world state, checking action preconditions, decomposing the goal). The point is not a better leaderboard number. It is a per-problem diagnosis that selects the repair. We test whether that diagnosis beats always applying the single best repair at the same compute budget.

Core Claim

The standard semantic-dependency measurement, the accuracy drop under obfuscation, plays the role of an outcome prediction in Liu et al.'s sense. It is informative about failure but has low action value, meaning it contributes little to choosing the correction that actually flips a failing problem. Liu et al. prove that predicting the outcome is enough for choosing an action only when one "sufficient action" works across all hidden states. Otherwise some pivotal measurement of the changeable hidden state has strictly higher action value.

We hypothesize that obfuscated planning failures are heterogeneous. Some problems fail because the model loses track of the world state, some because it misapplies preconditions it can no longer infer from word meaning, and some because the plan is too long. If so, no sufficient repair exists. A cheap factorial probe of each problem then acts as the pivotal measurement and selects better repairs than a failure predictor or any single fixed repair.

The claim is falsifiable. If one repair (for example, an explicit state ledger) recovers nearly everything, then a sufficient action exists, and that is itself a useful and reportable finding.

Research Program

The first component is a fresh, fully crossed generator of planning problems built on PlanBench's Blocksworld domain (Large Language Models Still Can't Plan).

Problem factors are varied at two levels each:

  • Action names: familiar, or obfuscated.
  • Predicate names: familiar, or obfuscated.
  • Token form of the obfuscation: real English words reassigned to new meanings, as in Mystery Blocksworld, or random strings that fragment into rare sub-word pieces. This factor is the explicit test of the tokenization question that LINGOLY-TOO argued against but did not cross with logical structure.
  • Plan length: the 3–5 block regime, or the 20–40 step regime where o1 drops to 23.6%.
  • Provenance: one of the 600 published instances, or a newly sampled instance with an identical solution structure.

The provenance factor treats contamination as a measured cause rather than a nuisance. If memorization matters, it should show up as a provenance effect that appears only together with familiar names. Fresh instances are regenerated per evaluation with new random name maps, following the dynamic-generation logic of DyVal.

Process switches are the analogue of the "longwave-off" experiments in CFMIP. Those experiments make clouds transparent to longwave radiation to isolate one physical pathway. Each of our switches removes one cognitive burden while holding everything else fixed:

  • State ledger: the true world state is injected after every proposed action.
  • Precondition checklist: each action's preconditions are restated in the obfuscated vocabulary at the point of use.
  • Subgoal decomposition: the goal is handed over as an ordered list of subgoals.
  • Name restoration (diagnostic only): an oracle glossary maps the obfuscated names back to their meanings.

The first three are realistic repairs that a user without the hidden name map could apply. The glossary is an upper bound used only for attribution.

Effects are estimated as in climate attribution, where the residual interaction term is the object of interest rather than an inconvenience (AerChemMIP2). We fit

$$\text{logit}\, P(\text{valid plan}) = \beta_0 + \sum_j \beta_j x_j + \sum_{j<k} \gamma_{jk} x_j x_k,$$

where each $x_j \in \{0,1\}$ indicates one factor or switch, $\beta_j$ is its main effect, and $\gamma_{jk}$ captures non-additivity. A large $\gamma$ between obfuscation and plan length, for instance, would mean that losing word meaning hurts mainly by breaking long-horizon state tracking.

The second component turns attribution into a decision. At deployment we cannot run all cells for every problem, so the probe is a small fractional design. For each problem we spend a fixed budget of a few short runs with individual switches on, observe which switch flips the trajectory toward validity, and then commit the remaining budget to that repair. We compare this policy against outcome-based and fixed-repair policies at matched total tokens.

The third component tests transfer to a second substrate where formal solvers are less natural. We use DyVal logic tasks relabeled with familiar versus arbitrary predicate words, with switches for supplying intermediate node values and reordering the description topologically. This checks whether the heterogeneity finding is specific to planning.

Models are one open-weight non-reasoning model, one open-weight reasoning model, and an API reasoning model on a reduced subset. The subset is necessary because o1-preview cost $42.12 per 100 PlanBench instances in the cited evaluation.

Baselines

Naive reference. The model receives the obfuscated problem with standard step-by-step prompting and no repair. Its accuracy on our fresh instances anchors the scale. The published figures are 52.8% for o1-preview on Mystery Blocksworld, 2.0% for GPT-4 on obfuscated Blocksworld, and "near-zero" for vanilla models (LLMs Still Can't Plan; Can LRMs?; Large Language Models Still Can't Plan). We re-measure rather than reuse these numbers, because our instances are freshly generated.

Best single repair. This is the strongest competitor and the direct test of Liu et al.'s sufficient-action condition. We choose the one repair, or fixed combination of repairs, that maximizes average accuracy on a held-out tuning split, then apply it to every problem. It receives the same total token budget as our policy, with leftover budget spent on extra samples. Beating it means a statistically reliable gain in valid-plan rate on the test split at matched tokens.

Outcome-predictor policy. This baseline corresponds to "prediction without actionability." A classifier predicts failure from instance features and the model's first-attempt trace, and the policy applies the best single repair only to problems predicted to fail. It operationalizes the claim that knowing who will fail is enough.

Self-consistency at matched budget. The model samples multiple plans and takes the most common valid-looking answer. This answers the objection that the probe's benefit is just extra compute.

Verifier back-prompting. An external plan validator returns error messages to the model. This is the lightweight version of the external-solver approach recommended by Valmeekam et al. We found no reported result on Lacuna for verifier-guided or repair-guided recovery on Mystery Blocksworld, so this baseline is newly measured.

Oracle selector. For each problem, this selector picks whichever repair actually works, using knowledge of the outcome in hindsight. It is the ceiling for any selection policy.

Ablation of the key ingredient. We replace the probe's observed switch responses with randomly permuted ones while keeping the same budget. This isolates whether the per-problem diagnosis, rather than probe compute, drives any gain.

Why not just…?

Why not just apply the best single repair everywhere? This is the real threat, and the proposal is built to settle it. Liu et al. show that a fixed policy is optimal exactly when a sufficient action exists, and nothing in the file says whether one exists for obfuscated planning. Our decisive statistic is the fraction of the oracle selector's gain over the naive reference that the best single repair captures. If that fraction is roughly 90% or more, we conclude that a sufficient repair exists and report that result instead of the selection claim.

Why not just hand the problem to a symbolic planner? For PDDL Blocksworld this is correct, and we say so plainly: Fast Downward solves these instances in milliseconds. We include verifier back-prompting as a baseline for that reason. The question here concerns settings without a complete formal domain, the setting Liu et al. and The risks of risk assessment describe. There, the model's own reasoning must be steered. Planning is used because it gives exact validity checks for measuring the effect of each switch. The DyVal transfer arm tests whether the conclusions travel.

Why not just run the obfuscated-versus-standard comparison, as PlanBench already does? That comparison yields one difference per model. It cannot separate token fragmentation from lost word meaning from depth, and it cannot detect interactions between them. The tension between LINGOLY-TOO, which finds tokenization is not the cause, and Can Large Language Models Reason and Plan?, which argues models rely on word co-occurrence, remains unresolved because nobody has crossed token form with logical structure. The paired comparison also produces only a prediction of failure, not a repair.

Why not just train a better failure predictor? Liu et al.'s Theorem 3 suggests that once the changeable state is measured, adding the outcome prediction rarely helps. The outcome-predictor baseline tests this directly. If it matches the probe policy, our pivotal-measurement framing is wrong for this domain.

Proposed Experiments and Decisive Checks

The attribution study runs the full crossed design on 200 fresh problems per cell for open models, and on a reduced fractional design for the API model. It reports main effects and interaction terms with bootstrap intervals.

Three hypotheses are each falsifiable:

  • Provenance acts through familiarity. The provenance effect is nonzero only together with familiar names. If memorization shows up equally under obfuscation, the contamination account is wrong.
  • Word meaning and plan length interact. The interaction between obfuscation and plan length exceeds the sum of their separate effects, mirroring the non-additivity known from climate attribution.
  • Token form is not the main cause. The token-form factor is small relative to the meaning factor. Finding the opposite would overturn LINGOLY-TOO's conclusion for planning.

The decision study compares the probe policy against all baselines in valid-plan rate per thousand tokens. It also reports the fraction of the oracle selector's gain each policy captures. For unsolvable instances, it reports correct unsolvability detection, against the published o1 figures of 27% detection and 54% confident wrong plans.

The work succeeds if the probe policy beats both the best single repair and the outcome predictor at matched budget on PlanBench, and the advantage survives on the relabeled DyVal tasks. It fails informatively if a single repair captures nearly all of the oracle selector's gain. That outcome would tell practitioners exactly which repair to always use.