When a team of language-model agents writes code to clean, merge, or deduplicate a table, the code usually runs but is often wrong. On the DataGovBench data-governance benchmark, many models produce code that executes more than 80% of the time while still violating the logic of the task (DataGovBench). The fix people reach for is a reviewer agent: something that runs the code and sends error traces back to the author. But an error trace only reports crashes. It says nothing about the thousand rows that silently broke a "no two records share an email" rule. We propose a reviewer that works differently. The task's policy is stated once, abstractly, as a small set of first-order rules. The reviewer then checks the pipeline's output against those rules the way lazy solvers in logic programming do. It never expands a rule over every combination of rows. It asks the database only for the specific row combinations that violate it, and hands those concrete counterexamples to whichever party should fix them.
The organizing picture comes from how journal editorial boards divide labor. An editor-in-chief sets policy. Associate editors triage each problem to the right place. Reviewers point at specific defects rather than issuing verdicts. The mechanism comes from lazy grounding. "Grounding" is the step where a solver substitutes every combination of concrete objects into an abstract rule, which blows up as $O(N^k)$ for a rule with $k$ variables over $N$ objects. Lazy grounding does this substitution only for the combinations that currently matter. The claim is that feeding violated ground instances back to agents, and routing them by how many there are, closes much of the gap between code that runs and code that is correct. It should do so at a cost that stays small as tables grow.
Core Claim
A three-tier review loop closes more of the runnable-versus-correct gap on multi-step data-governance tasks than a sandbox evaluator or an LLM judge, and its memory and token cost scale with the number of violations rather than with table size. The three tiers are:
- A policy tier that translates the task description into lifted, meaning variable-bearing, first-order constraints.
- A separation tier that queries the current output for violated instances of each constraint.
- A triage tier that routes each constraint's violations. When violations are few and scattered, it sends them to a lazily grounded repair solver that makes minimal edits. When they are many and share a pattern, it returns a handful of counterexamples to the code-writing agent, because the code itself is wrong.
The new ingredient is this triage between solver repair and code revision. It is driven by violation counts obtained without full grounding. Neither the lazy-grounding literature nor the multi-agent literature does this.
Relation to Prior Work
The mechanism is borrowed from three lazy-grounding results.
- Cutting Plane Inference (Cutting Plane MAP Inference for Markov Logic) solves a partial problem, finds violated constraints through database queries (a step called "separation"), and adds only those. On citation matching it cut active ground formulas from 2 million to about 19,000 and reached an exact solution in under two minutes, where full integer programming could not even be initialized. We reuse its separation-by-query step almost verbatim. Our separation oracle is SQL over the agent's output table, and our repair solver is Cutting Plane Inference over integer programming.
- Lazy model expansion (Lazy Model Expansion: Interleaving Grounding with Search) keeps "justifications," symbolic reasons a rule is satisfied without expanding it. It grounds only the instances whose justification breaks, which cut one benchmark from nearly 30 million atoms to 16,000. It also states the caveat that shapes our design: a lazy solver is less informed and can thrash.
- Lazy normalization (Enhancing Lazy Grounding with Lazy Normalization in Answer-Set Programming) handles counting constraints, such as "at most three addresses per customer," without knowing domain sizes in advance. It scaled to 2,000 time steps in 80–620 MB, while the full-grounding solvers Clingo and Dlv2 crashed at 600 steps against a 40 GB limit. It supports only monotone aggregates, which limits which policies we can encode. We scope the constraint language accordingly and report coverage honestly.
None of these systems has an agent in the loop. None of them must decide whether a violation is better fixed by editing data or by rewriting the program that produced it.
The multi-agent side supplies the setting.
- DataGovAgent (DataGovBench) already uses a Planner that writes pre- and post-condition "contracts" over a task graph, an Executor that writes code, and an Evaluator that runs it. The Evaluator cut debugging iterations from 14.89 to 3.29. However, its contracts are procedural and checked by execution, which is why the runnable-but-wrong failure persists. Our proposal replaces that Evaluator. The Planner and Executor stay as they are.
- Organized LLM teams (Embodied LLM Agents Learn to Cooperate in Organized Teams) show that imposing hierarchy on LLM teams improves efficiency and keeps communication linear in team size. The same work concedes that agent behavior cannot be formally bounded. This motivates us: our loop has a checkable stopping condition, namely zero violated instances of the stated policy.
- FirstPass (FirstPass) shows that editorial judgment lives in tracking unresolved concerns across rounds. Only with a specific training scheme does its model predict revision length well, at 80.5% accuracy. We take the analogy literally. Our reviewer keeps a ledger of violated ground instances across rounds, and the ledger is the "unresolved concerns" list. FirstPass learns these concerns statistically from text; we compute them.
- LLM-plus-ASP planning (Improving Efficiency of Answer Set Planning with Rough Solutions from Large Language Models) passes LLM rough solutions to an answer-set solver once, in sequence. It neither grounds lazily nor iterates between tiers.
Research Program
The policy tier. An LLM receives the natural-language task and the input schemas. It emits constraints in a restricted language: universally quantified clauses over table rows with equality, comparison, and monotone count and sum aggregates. An example is "for all rows $r, s$ with $r \neq s$, not (email($r$) = email($s$))". Each constraint compiles to one violation query. Because translation errors would confound everything downstream, we run every experiment twice: once with LLM-written constraints and once with constraints written by hand for a subset of tasks. The gap between the two measures specification error separately from review quality.
The separation tier. For each constraint $c$, this tier runs the violation query on the current output and returns the violated set $V_c$ together with its size. It never materializes all $O(N^k)$ instances. Its cost is dominated by the query, which a database can often answer with joins on indexed columns. That is the same trick lifted planners use when they generate successors as conjunctive queries (Lifted Planning).
The triage tier. For each constraint, the triage agent sees $|V_c|$, a sample of violations, and the provenance of the offending rows, meaning which pipeline step produced them. It chooses between two routes:
- Solver repair. This handles scattered violations. It is Cutting Plane Inference over integer programming that finds a minimal set of cell edits or row deletions satisfying all constraints, grounding only the constraints violated at each round.
- Code revision. This handles systematic violations. The Executor receives at most a few counterexamples plus the lifted rule.
We will compare a learned or prompted triage policy against a fixed threshold rule on $|V_c|/N$. The threshold rule may turn out to be enough, and that would be a useful finding in itself.
Termination. The loop ends when every $V_c$ is empty or a round budget is spent. We log instances grounded, peak memory, tokens, and rounds.
Baselines
The naive reference point. This is a general-purpose code agent with no structured evaluator. DataGovBench reports a DAG-level Average Task Score of 39.7 for this setting (DataGovBench). We re-run it with our backbone model, so that 39.7 serves as a sanity anchor rather than the comparison.
The strongest existing method. This is DataGovAgent's Planner–Executor–Evaluator assembly line, reported at an Average Task Score of 54.9 on the DAG tasks, with a 64% success rate when paired with GPT-5 (DataGovBench). It is the right competitor because it already has hierarchy, contracts, and an iterative evaluator; only our reviewer differs. For a fair comparison we keep its Planner and Executor, the same backbone model, the same retrieval corpus, and the same maximum number of review rounds and total token budget. The only change is the Evaluator. Beating it means a higher Average Task Score and success rate on the 50 DAG-level tasks under the matched budget, measured with paired per-task comparison. We also need a lower rate of outputs that are runnable but fail the task's logic. A gain in the score alone, with no drop in that failure category, would not support the claim.
The full-grounding ablation. This variant uses identical constraints and repair, but grounds all constraint instances up front and solves the repair with one integer program. It tests whether laziness matters. On small tables it should match our accuracy, and it should fail on memory as rows grow.
The key-ingredient ablation. This variant uses the same separation tier but always routes violations to the Executor, with no solver repair. It is essentially an assertion-based evaluator. A second variant always routes to the solver. Together they isolate whether triage is the thing that helps.
The LLM-judge reviewer. This variant replaces the separation tier with an LLM that reads the output sample and the policy text and lists problems. It uses the same round and token budget.
Why not just…?
Why not just write assertions and let the Executor retry? This is the most serious threat. An assertion that runs the violation query and returns failing rows is most of our separation tier. If always-route-to-Executor matches the full system, then the lazy repair solver and triage add nothing, and the contribution reduces to "state the policy as queries." We would report that result as such. Our expectation that it falls short rests on two points. First, some violations are data problems, not code problems: residual near-duplicates, or conflicting sources in integration tasks. Rewriting code does not reliably fix these, and retry loops are exactly where the 14.89-iteration behavior came from. Second, a minimal repair that respects all constraints jointly is a combinatorial problem that an LLM editing code cannot guarantee. The ablation settles the question directly.
Why not just ground everything and hand it to Clingo or an integer-program solver? On DataGovBench-sized tables this may work fine, and the full-grounding ablation will show that. The reason not to rely on it is scaling. Pairwise constraints like deduplication grow quadratically in rows. Full grounding already failed at 40 GB in answer-set planning (Lazy Normalization) and could not initialize on citation matching (Cutting Plane Inference). Our row-scaling experiment is designed to find the crossover point, and to show whether one exists at realistic table sizes.
Why not just use a stronger LLM as the reviewer? Organized-team work concedes that LLM behavior cannot be bounded (Embodied LLM Agents). FirstPass found that without its specific training scheme, accuracy fell to 62%, which was worse than guessing the majority class (FirstPass). A judge reading a sample of rows cannot see a violation hidden among a million pairs. The LLM-judge baseline measures how much this costs on DataGovBench. If a strong judge closes the gap on small tables, we expect it to lose on the scaled ones.
Proposed Experiments
Main comparison. The primary benchmark is DataGovBench-150, with emphasis on its 50 DAG-level multi-step workflows. We first report coverage: the fraction of tasks whose correctness criteria can be expressed in our constraint language. Monotone-aggregate restrictions and fuzzy tasks such as classification will exclude some. Results are reported on the covered subset and on the full set, with uncovered tasks falling back to the DataGovAgent Evaluator. Metrics are:
- Average Task Score and success rate.
- The runnable-but-incorrect rate.
- Review rounds.
- Tokens.
- Ground instances created and peak reviewer memory.
Scaling test. DataGovBench tables are small, so we build scaled versions of the deduplication, integration, and imputation tasks by growing row counts by factors of 10 to 1,000 while preserving the constraint structure. We then compare the lazy reviewer, full grounding, and the LLM judge. There is no reported result found on Lacuna for any method on scaled DataGovBench tables, so every number here is new. As an external check that the separation tier works as a cutting-plane oracle, we reproduce the citation-matching setting from Cutting Plane Inference. There we expect active constraints in the tens of thousands rather than millions, consistent with its reported 2 million to about 19,000 reduction.
Decisive checks. Four results would determine what the project has shown.
- Does the core claim hold? The core claim fails if the full system does not beat DataGovAgent's Evaluator on the runnable-but-incorrect rate under a matched budget.
- Is the contribution only "policies as queries"? The contribution shrinks to that if always-route-to-Executor matches the full system.
- Does laziness matter? Laziness is unnecessary for this domain if full grounding keeps pace at 1,000× rows.
- Is specification the bottleneck? The bottleneck is specification, not review, if hand-written constraints beat LLM-written ones by more than the gap between our reviewer and the baseline. In that case the next project is the policy tier.
References
- DataGovBench: Benchmarking LLM Agents for Real-World Data Governance Workflows
- Cutting Plane MAP Inference for Markov Logic
- Lazy Model Expansion: Interleaving Grounding with Search
- Enhancing Lazy Grounding with Lazy Normalization in Answer-Set Programming
- Embodied LLM Agents Learn to Cooperate in Organized Teams
- FirstPass: Grounding AI Scientific Judgment in Multi-Round Editorial Outcomes
- Resolving Word Vagueness with Scenario-guided Adapter for Natural Language Inference
- Lifted Planning: Recent Advances in Planning Using First-Order Representations