Blame-Routed Repair: Fixing LLM-Written Game Rules with Planner Playouts

Language models can now write small video games in VGDL, the Video Game Description Language: a short declarative text that lists the game's objects, what happens when two objects touch, how the game ends, and how characters in an ASCII map correspond to objects. The hardest mistakes are not syntax errors. They are files that parse and run but describe the wrong game. The best-known example is an argument-order slip. Natural English says "the avatar reaches the goal," so the model writes avatar goal > killSprite. In VGDL that line deletes the player, not the goal (Generating Games via LLMs). A compiler cannot catch this. A game-playing search agent catches it within a few moves, because the player dies the instant it reaches the goal. The engine's event log even names the rule that fired.

This note proposes using that log as the repair signal. A fast tree-search player probes each generated game. Each observed failure is traced to the specific VGDL block and line that caused it, and only that block is handed to a repair step, which edits it in isolation and can be rolled back. The bet is that VGDL's own four-block structure is the right way to split the work among agents, better than generic "analyst, coder, tester" roles. It is also the right granularity for checkpoints, which should prevent the familiar pattern where a fix introduces a new bug and debugging stalls after two or three rounds.

Core Claim

Behavioral bugs in generated game code can be repaired reliably only when two conditions hold. The feedback must come from executing the game with a planner, not from parser messages or from the model reading its own code. That feedback must also be attributed to the responsible rule and routed to an editor whose scope is limited to that rule's block. The claim has three separable parts, and the experiments test each one. First, planner playouts fix errors that parser feedback cannot. Second, attribution matters beyond simply reporting "the game is unwinnable." Third, block-scoped editing with per-block rollback beats a single agent that sees the same attributed diagnostics. If the third part fails, the contribution narrows to attributed playout feedback, and the note says so in advance rather than defending a multi-agent design for its own sake.

Where This Sits

The closest prior work stops short in specific, documented ways. Generating Games via LLMs (LLMGG) generates VGDL in one open-loop pass and identifies the natural-language ordering bias. It names playability testing in a loop as future work, which this proposal builds. GAVEL evolves Ludii board-game code by using a fine-tuned model as a mutation operator. It discards broken offspring instead of repairing them and reports many unused game components. Our dead-rule diagnostic targets exactly those components. From Gameplay Traces to Game Mechanics shows that factoring a game into entity, dynamics and observation layers before writing VGDL reduces hallucinated rules. That pipeline is feed-forward and never runs the result. We borrow its insight that layered structure helps, but apply it at repair time with an executor in the loop.

On the refinement side, MEIC supplies the method we reuse most directly: it combines a debug agent, a scorer, toolchain feedback and rollback to the best previous version. Its oracle is a hardware testbench, and games have no testbench. CodeTransOcean supplies the constraint. Its self-debugging gains are front-loaded, rising from 48.57% to 52.57% over three rounds. Our iteration curves must beat that plateau shape, not just the starting point.

For multi-agent design, Self-collaboration Code Generation and MAGIS show that role-split teams help on general code. Adversarial Review warns that adding agents gives diminishing returns. Self-collaboration also reports that models under about 13B parameters fail to hold persona boundaries. This last finding motivates splitting by context, where each editor sees only its block, rather than by persona. We test explicitly whether that split holds up for 8B models.

Finally, GVGAI-LLM reports that language models play VGDL games badly (GPT-4o-mini wins 10.27% across 118 games) while tree search reaches 100% on comparable symbolic tracks. Verification therefore belongs to a planner, not to the language model.

Research Program

The system has three parts.

The first is an instrumented VGDL runtime built on the GVGAI engine (GVGAI multitrack framework). For every collision event and termination check, it logs the rule identifier that fired.

The second is a prober. It runs Monte Carlo tree search (MCTS), plus random and breadth-first-search rollouts, for a fixed budget. It then emits typed diagnostics, each tied to a line. An avatar-destroyed-by-rule diagnostic points to the InteractionSet line. A termination-never-reachable diagnostic arises when no rollout triggers any win condition and points to the TerminationSet. A goal-unreachable diagnostic, found by search over the map with mechanics held fixed, points to the LevelMapping and the map. A dead-rule diagnostic flags interaction lines that never fire across all rollouts. Parser and runtime exceptions are attributed by line as usual.

The third part is the editors. Each one owns a single block (sprites, interactions, termination, or mapping plus map). An editor receives the design brief, a read-only view of the other blocks, and only the diagnostics attributed to its own block. A coordinator applies each edit, re-probes the game, and keeps a per-block checkpoint. Any edit that increases the weighted diagnostic count is reverted, in the style of MEIC but at block granularity.

The design has one honest hazard: a repair loop driven by playability can game the signal by producing trivially winnable games. For example, it could delete enemies, or define a win condition that is already satisfied at step zero. Two measures address this. In the loop, the prober flags wins in fewer than k steps and flags rule deletions that are not justified by a diagnostic. The main evaluation uses ground-truth behavior that the loop never sees.

The evaluation task is built from games with known answers. GVGAI-LLM already converts VGDL games into natural-language descriptions. We use those descriptions as briefs and ask systems to reconstruct the games, holding out the original VGDL for scoring. We add a corruption track modeled on MEIC's error set. We inject realistic bugs into original GVGAI games, including argument swaps, wrong sprite classes, missing termination conditions and mis-mapped map characters. This yields a fix-rate measure with exact ground truth.

Baselines

The naive reference point is one-shot generation with the full grammar-plus-example prompt from LLMGG (its P7 configuration). On its Maze task, LLMGG reports 10/10 correctness for GPT-4, a peak of 4/10 for GPT-3.5, and 0/10 for Gemma 7B (Generating Games via LLMs). We found no reported result on Lacuna for VGDL reconstruction across the GVGAI suite. The Maze numbers show that one easy game is solved at the frontier, not that the problem is.

The second baseline is single-agent self-debugging with parser and runtime errors only. This is the standard execution-feedback loop that CodeTransOcean measures, and it isolates the value of planner playouts.

The strongest competitor, and the key ablation, is a single agent that receives the identical attributed playout diagnostics and the same rollback rule but edits the whole file. Beating it is the only evidence that block-scoped decomposition matters.

The fourth baseline is best-of-N sampling with the prober used purely as a filter, run at a matched token budget. It tests whether repair beats resampling.

The fifth is a Self-collaboration-style analyst, coder and tester team (Self-collaboration Code Generation) in which the tester is a language model with no planner. It represents the generic multi-agent recipe.

The sixth, scored on the corruption track, is a GAVEL-style mutate-and-discard loop: unguided model edits kept only if the diagnostic count improves. It represents evolution without reading the error.

All systems use the same base models (an 8B open model, a roughly 32B open model, and one frontier model), the same briefs, the same prober budget, and a fixed cap on total generated tokens per game. The single prompt template for each baseline is tuned on a shared development split of 15 games. "Beating" a baseline means a statistically significant gain in behavioral agreement on held-out games under a paired test across games and three seeds. Playability alone does not count.

Why not just…?

Why not just give one strong model the engine's error messages? Parser messages are silent on the bugs that matter. avatar goal > killSprite is valid VGDL. The parse-only baseline against the planner-feedback arm, scored on the subset of corruptions that parse cleanly, measures this directly. We expect parse-only repair to fix syntax faults at high rates, as MEIC reports for syntax, and to leave semantic faults largely unfixed.

Why not just sample many games and keep the ones the planner can win? This is a real threat when the base model is strong and briefs are short, because the right answer may appear among the samples. It weakens when a game needs several independent blocks to be correct at once, since the chance that one sample gets all of them right shrinks multiplicatively. We report agreement as a function of the number of interaction rules in the original game, at matched tokens. If best-of-N matches repair even on the most complex games, the repair loop is unjustified and we will say so.

Why not just give one agent the attributed diagnostics, with no team? This is the most likely way the multi-agent part of the claim fails. Adversarial Review gives a reason to expect it might. The single-agent-with-attribution arm is run at equal budget specifically to settle the question. The prediction that would favor decomposition is concrete: whole-file editing should show a higher regression rate, meaning previously correct blocks get broken. That gap should be largest for the 8B model, where long-context edits drift most. If no regression gap appears, the paper reports attributed playout feedback as the contribution and drops the team.

Why not just evolve games as GAVEL does? Discarding a broken offspring throws away the information in its failure. The corruption track compares fix rate per model call for guided repair against undirected mutate-and-keep.

Proposed Experiments

The reconstruction study uses the GVGAI single-player games with GVGAI-LLM's descriptions as briefs. Its main metric is behavioral agreement: replay matched action sequences (random, plus MCTS traces collected on the original game) for H steps in the original and generated games, and score the fraction of steps whose object-level states match after aligning sprite names. Secondary metrics are parse rate, MCTS win rate on the generated level, the fraction of dead rules, the regression rate per iteration, and tokens used.

The corruption study reports fix rate split into syntax faults and semantic faults, mirroring MEIC's split.

Four decisive checks follow from the claims. The planner-versus-parser comparison on semantic faults tests whether execution feedback is necessary. An arm that gives the model only a summary ("unwinnable; player died at step 7") without naming the rule tests whether attribution is necessary. The block-scoped versus single-agent comparison, analyzed by model size, tests whether decomposition is necessary. Agreement plotted over iterations one through eight tests whether block-level rollback avoids the two-to-three-round plateau seen in CodeTransOcean.

A final check covers reward hacking: for each system, the share of games that are playable but low in agreement. A system that raises win rate while lowering agreement has learned to satisfy the prober, not to build the requested game, and is counted as failing.