Arguing With the Critic: Renderer-Grounded Argumentation for Visual Code Refinement

When a language model writes code that draws something, such as a Matplotlib figure or a web page, the standard fix for its visual mistakes is a loop. The code is rendered to an image, a vision-language model (a model that reads images and writes text) critiques the image, and the code model revises. These loops help, but they fail in a recognizable way. Critiques are sometimes wrong, sometimes contradict each other, and the reviser often "fixes" something that was already correct. The loop also stops improving after two or three rounds. This proposal treats each critique as a defeasible claim: it stands unless something knocks it down. The things that can knock it down are exact geometric measurements taken from the renderer itself, such as bounding boxes, overlaps and clipping. A small logic solver then decides which critiques survive, which earlier successes must be protected, and which questions are worth asking the user.

The two-sentence version is this. Instead of pasting free-text critiques back into the prompt, we compile critiques, prompt requirements and renderer measurements into a structured argumentation framework, and solve it with an answer-set solver. Only critiques that survive go to the code model, and edits that break previously established requirements are rejected. The bet is that this makes refinement more accurate and also monotone: later rounds do not undo earlier wins, and questions to the user target only the genuinely unresolved points.

Core Claim

Formal argumentation has a well-developed, exact link to logic programming. Schulz and Toni showed that the 3-valued stable models of a logic program correspond one-to-one to the complete labellings of its Assumption-Based Argumentation (ABA) counterpart, and that answer sets correspond to stable extensions (Logic Programming in Assumption-Based Argumentation Revisited, From Logic Programming to Argumentation and Back). ABA is a form of argumentation where conclusions rest on explicit assumptions that other derivations can attack. A complete labelling marks every argument as in, out, or undecided. Because of this correspondence, we can state the dialectics of a refinement loop as a few dozen ground rules and get accepted sets and undecided sets from an off-the-shelf answer-set solver in milliseconds.

The claim is specific. Visual refinement loops degrade because they have no persistent, adjudicated record of what is known about the artifact. Two things supply that record:

  1. A non-neural source of attacks: measurements read from the render tree (DOM boxes in a headless browser, artist extents in Matplotlib).
  2. Stable and complete semantics to settle what is currently accepted.

With this record in place, the loop should stop regressing, should become robust to wrong critiques, and should limit clarifying questions to undecided arguments.

How It Relates to Prior Work

The closest prior work is CAELF (My Grade is Wrong!). CAELF maps multi-agent essay judgments into a Dung abstract argumentation framework and re-evaluates it when a student contests a grade. It keeps an 80–90% "Maintain Truth" rate, while prompted LLMs fall below 40%. CAELF motivates our proposal because it shows that argumentation semantics resist capitulating to pushback. It differs from our proposal in three ways:

  1. It is text-only.
  2. Its arguments are opaque nodes whose attacks are themselves judged by an LLM.
  3. Nothing in it is measured.

Our attacks on visual claims come from deterministic renderer geometry, so some conflicts are settled by fact rather than by another model's opinion.

SceneCraft (SceneCraft) is the closest work on the vision side. It compiles scene-graph relations into numerical constraints and runs a GPT-4V critic-and-revise loop. It reaches 88.9% constraint passing against BlenderGPT's 5.6%. Its constraints are hard, and conflicts between critiques and requirements are handled by re-prompting. We keep its idea of a symbolic blueprint, but make requirements defeasible and make the critiques themselves attackable.

Several papers motivate the gap:

  • MatPlotAgent (MatPlotAgent) shows that visual feedback matters at all: its GPT-4 score falls from 61.16 to 53.44 without the visual agent.
  • Sketch2Code (Sketch2Code) documents the plateau after 2–3 rounds and the degrading proactive questions.
  • FIRE (FIRE) documents edits to correct code under ambiguous feedback.
  • R³ (Reason-Reflect-Rectify) documents that models detect flaws far better than they repair them.

Volcano (Volcano) shows that critiquing increases visual attention. This supports keeping a vision-language critic in the loop rather than replacing it.

The method draws on two further sources. The ABA–logic programming correspondence supplies the encoding. Constrained argumentation frameworks (Abstract argumentation frameworks with strong and weak constraints) supply weak constraints, which let us rank requirements by stated user priority. Schulz and Toni's attack-tree explanations (Explaining Answer Set Programming in Argumentative Terms) supply the format for telling the code model why a critique was accepted.

Research Program

Requirements. Each task begins with requirement extraction. An LLM turns the prompt, or the target screenshot in Design2Code, into typed requirement atoms. Some are geometric ("legend does not occlude data", "nav bar is above hero image", "no text clipped"). Others are semantic ("bars are colored by group", "title reads X"). Geometric atoms come with a checker that runs on the render tree. Semantic atoms do not.

Encoding each round. After each render, the vision-language critic emits claims of the form "requirement r is violated" or "requirement r is satisfied", and these enter the framework as assumptions. Each checker result enters as a strict fact. The contraries are wired as follows:

  • A measured satisfaction of r is the contrary of the assumption "r is violated".
  • A measured violation is the contrary of "r is satisfied".
  • Semantic claims have no measured contrary, but can be attacked by a contradicting critic claim from a second sample or a second critic.

Edits. A proposed code edit is an assumption "apply e". The edit is attacked if the counterfactual render under e yields a measured violation of any requirement that was accepted as satisfied in the previous round. This is how regressions become explicit and rejectable.

Solving. Clingo computes the stable extensions, plus the grounded and complete labellings when no stable extension exists. Over-constrained cases, such as two requirements that cannot both hold geometrically, produce exactly such odd cycles.

What the code model receives. The reviser gets only the critiques that are accepted, plus a short attack-tree justification for each. It never sees critiques that were defeated. Arguments labelled undecided become the only material from which clarifying questions may be drawn. This is our direct test of the Sketch2Code proactivity finding.

Cost. The added cost is one extra render per candidate edit plus solver time, which is negligible for frameworks of this size. Volcano's latency doubling (2.7 s to 5.8 s) is the reference point for reporting overhead honestly.

Baselines

Do-nothing reference. The do-nothing reference is single-pass generation with the same code model and no feedback. It anchors every gain.

Strongest existing method. The strongest existing method is the MatPlotAgent-style loop (MatPlotAgent): a code agent with interpreter debugging plus a vision-language visual agent whose free-text critique is fed back. On MatPlotBench it reports 61.16 with GPT-4, against a 48.86 single-pass baseline. On Design2Code we use the paper's own self-revision prompting. It reports a GPT-4o Block-Match of 93.0 on the full set and 56.6 on Design2Code-HARD (Design2Code). Those figures come from different model versions than ours will, so we re-run the loop ourselves.

Fairness. We re-implement the loop with the identical code model, critic model, number of rounds (five), prompt-tuning budget, and total number of renders. The renders matter because our method spends renders on counterfactual edit checks. The loop gets the same count, which it can use as extra critique samples. Beating it means a higher final MatPlotBench score and Block-Match, and a lower regression rate, at matched renders.

Ablation 1: measure-and-filter. This uses the same renderer checkers but no argumentation. Any critique contradicted by a measurement is dropped, all others are passed through, and there is no edit rejection. It isolates whether gains come from geometric grounding alone.

Ablation 2: checklist with regression test. This accumulates requirements that previously passed and rejects any edit whose render fails one, using the same checkers. It approximates the protective effect without stable or complete semantics. Neither ablation can distinguish undecided from rejected arguments, conflicts among semantic critiques, or priority trade-offs.

Ablation 3: no geometric checkers. This is the full argumentation framework but with LLM-judged attacks only, which is the CAELF-style variant. It isolates the value of non-neural attacks.

What would make the formal machinery worth keeping. The full method must beat Ablations 1 and 2, specifically on items where critiques conflict or requirements are jointly unsatisfiable. If it does not, the formal machinery is not justified and we will report that grounding alone carries the gain.

Why not just…?

Why not just prompt a frontier model to self-revise more carefully? This is the strongest-baseline row above. Published evidence is against it as a complete answer. Sketch2Code shows feedback-following gains of at most 7.1% over five rounds and degradation in proactive mode. CAELF shows prompted models abandon correct positions under pushback. Our regression-rate metric measures directly whether careful prompting stops previously satisfied requirements from breaking. If prompted revision matches us on regression rate, our central claim fails.

Why not just fine-tune the model on feedback trajectories, as FIRE or R³ do? Fine-tuning raises refinement skill: FIRE reports a 50% gain in its self-correction metrics, and R³ reports +9.0% rectification. But FIRE notes that ambiguous feedback still causes hallucinated edits, and R³ needs a hierarchical reward to stop its policy from cheating. Our method is training-free and compatible with fine-tuning. We will run it on top of an open model both with and without FIRE-style tuning, to test whether the gains are additive or redundant. A weights-only fix cannot reject a wrong critique the model has no means to verify, and that is exactly the case our injected-noise test targets.

Why not just check renderer measurements and drop contradicted critiques, with no logic solver? This is the real threat, and it is Ablation 1. On single, independent critiques we expect it to match us. The formal semantics should matter only in three cases: when critiques interact, when an edit fixes one requirement while breaking another, and when requirements cannot all hold, where the undecided label decides what to ask. We will stratify results by whether an item contains such conflicts. If the advantage over Ablations 1 and 2 is absent even on the conflict stratum, we will drop the argumentation layer and report the simpler system.

Why not just use a numerical layout solver, as SceneCraft does? Numerical solvers need every constraint to be hard and every quantity to be numeric. Semantic critiques and user priorities, such as "label above bar unless it collides with the title", are neither. We include a SceneCraft-style variant on the geometric subset of requirements only, to show where it suffices.

Proposed Experiments

Benchmarks.

  • MatPlotBench: 100 cases, scored by its GPT-4V scorer, which the paper reports at 0.876 Pearson correlation with humans.
  • Design2Code and Design2Code-HARD: Block-Match and visual similarity.
  • Sketch2Code: both feedback-following and question-asking modes.

On top of task scores we add three metrics:

  • Regression rate: the fraction of requirements measured as satisfied at round t and violated at round t+1.
  • Round-wise monotonicity: whether score curves stop plateauing after round 3.
  • Question quality in Sketch2Code: redundancy and the gain per question.

Decisive check 1: robustness to wrong and conflicting critiques. We inject wrong critiques that a checker could falsify and contradictory critique pairs at controlled rates (10–40%). We then measure how often the critiques passed to the reviser are wrong, and how the final score holds up. The prediction is that our method's final score stays roughly flat as noise rises, while the free-text loop and Ablation 3 degrade.

Decisive check 2: the conflict stratum. This is the comparison against Ablations 1 and 2 on items with interacting or unsatisfiable requirements, as described above.

Decisive check 3: targeted questions. In Sketch2Code question-asking mode, we test whether restricting questions to undecided arguments removes the post-round-3 degradation without lowering final similarity. The comparison is against the paper's proactive baseline, for which no absolute per-round figures beyond the 7.1% feedback-following gain were found on Lacuna.

Reporting. We report latency and render counts for every comparison.