Text-to-image diffusion models generate images by gradually denoising random noise. They are good at rendering what a prompt describes, but they handle rules poorly. Ask for "exactly three cups, none next to a red object, the lamp always left of the sofa" and the model will often drop an object, blend attributes across objects, or silently ignore a negation. This proposal makes the rules a separate, checkable object. A domain expert writes constraints in a controlled natural language, meaning restricted English whose sentences map one-to-one onto formal logic. Those sentences compile into an Answer Set Programming (ASP) program, a declarative logic language whose solver enumerates every assignment that satisfies all the rules. The solver does three jobs. It reports before any GPU time is spent whether the request can be satisfied at all. It produces concrete object boxes and attribute bindings that region-guided diffusion then renders. After generation, it checks the image's detected scene graph against the same program and points to the exact rule that failed.
The two-sentence version for a colleague is this. Today's layout-guided diffusion methods take hand-typed coordinates or an LLM's guess at them, and the result is checked, if at all, by a vision-language model that can hallucinate. We replace both ends with one formal program that a non-logician can write, so that the planner and the checker are the same logic and a failed image can be traced to a broken rule and re-sampled from a different valid layout.
Core Claim
On prompts that carry relational or combinatorial structure, the claim is that a single compiled logic program used as satisfiability check, layout generator, and output verifier will raise the rate of images that satisfy every constraint, compared with LLM-planned layouts and with VLM-critic iterative refinement at matched compute. Relational or combinatorial structure here means exact counts, negations, adjacency and exclusion rules, and conditional placement. We expect the gap to grow with the number of interacting constraints and to be near zero on simple single-relation prompts. Those simple prompts serve as a control: if the method wins there, something other than the logic is doing the work.
This is different from the closest work in the file. Continuous Diffusion Models Can Obey Formal Syntax steers diffusion with exact gradients from finite automata. Its scope is one-dimensional token sequences and regular languages, and it does not cover spatial or combinatorial scene rules. Constraints-Guided Diffusion Reasoner for Neuro-Symbolic Learning takes the opposite route and internalizes fixed puzzle rules through reinforcement learning with a binary reward. It cannot accept a new rule at run time, which is exactly what an expert-facing interface needs. Compositional Neuro-Symbolic Reasoning shows that symbolic scene graphs plus consistency filtering beat frontier LLMs on abstract grids, but it synthesizes programs and never generates images. The layout-guidance line contributes the rendering step: MultiDiffusion fuses regional prompts, iLGD combines early attention injection with later loss guidance, and Diffusion Self-Guidance uses energies built from attention centroids. All three take coordinates as given. We reuse them unchanged as the renderer and supply what they lack, which is where the coordinates come from and how anyone knows they were obeyed.
Research Program
The first component is a scene controlled natural language built on the design of cnl2asp and CNL2ASP. Those tools compile five proposition types, including choices, hard constraints and soft constraints, into clingo programs, and they manage variables implicitly. That design is what lets a graph-coloring problem fit in seven sentences. We extend the vocabulary with object classes, qualitative spatial relations (left of, above, adjacent, inside, in front of, measured by depth), counts, and attribute predicates (color, texture, size class). Spatial relations are grounded on a discretized canvas grid, so the solver reasons over a finite domain. We also add a lint pass modeled on CNL-P, whose static analyzer caught all 47 error types it was tested on. The pass flags undefined objects and type mismatches before solving.
The second component is the solver loop. Before generation, an unsatisfiable program returns a minimal conflicting set of rules, rendered back to the user as the original sentences. A satisfiable program returns several diverse answer sets, each consisting of boxes, depth ordering, and per-object attributes. Soft constraints and an aesthetic prior, such as minimum object size and margins, rank the answer sets. Each answer set drives a renderer that uses iLGD-style box guidance with per-region prompts. Where an attribute must be bound to a single object, we add Token Merging-style composite tokens. In our setting those tokens come from the solver's explicit object–attribute pairs rather than from a SpaCy parse, which removes the parser-error failure mode that paper concedes.
The third component closes the loop. A detector plus a depth estimator turns the image into ground facts, using the UniDet-based pipeline that T2I-CompBench already uses for spatial and counting evaluation. Color facts come from mean Lab values inside each detected mask. The facts are asserted into the same ASP program. Any violated rule names specific objects and regions. On a violation, the system takes one of two actions: it re-renders only the offending region, or it switches to the next-ranked answer set.
The scientific questions are which of the three solver roles carries the gain and whether the CNL is actually usable by non-logicians.
Baselines
The naive reference point is the base diffusion model, SDXL or SD3, given the plain-English prompt, with best-of-N sampling at the same compute budget as our full loop. This measures how much any structure helps at all.
The strongest planning competitor is an LLM layout planner. A frontier LLM reads the same English request and emits boxes and attributes, which then go through exactly our renderer. Because the renderer is shared, any difference comes from planning. We will give the LLM few-shot examples and allow it to self-check its own boxes, using the same tuning budget we spend on CNL design. No reported result for LLM-to-box layout planners was found on Lacuna, so we will establish this number ourselves.
The strongest end-to-end competitor is VLM-critic iterative refinement as in Iterative Refinement Improves Compositional Image Generation. It reports a 16.9% improvement in all-correct rate on ConceptMix with $k=7$ concepts, a 13.8% improvement on T2I-CompBench 3D-spatial, and a Visual Jenga solve rate rising from 64.29% to 76.79%, all against compute-matched parallel sampling. We will run it with the same base model and the same total number of diffusion and judge calls. To beat it, we need a higher absolute all-constraints-satisfied rate on ConceptMix $k=7$ and on our constraint-heavy suite at equal compute. Our method must also not lose on T2I-CompBench's attribute-binding category. For reference, the strongest fine-tuning number the file reports there is GORS, which raised B-VQA from 0.5065 to 0.6603. Attention-only guidance is a further reference: Attend-and-Excite, plus ToMe, which scored 0.95 against SDXL's 0.49 on its GPT-4o binding set.
Three ablations isolate the key ingredient. The first replaces the ASP verifier with the VLM critic while keeping solver planning, which isolates exact verification. The second removes resampling of alternative answer sets, so failures can only be fixed by region edits. The third has an LLM translate free English into the ASP program instead of having the user write CNL, which separates the value of the solver from the value of the language.
Why not just…?
Why not just have a frontier LLM write the boxes? This is the main threat on easy prompts, and we expect to tie it there. The evidence that it degrades on rule-heavy requests is indirect. GPT-4o found logic errors with 76.6% accuracy and a 19% rate of invented errors in CNL-P. Frontier models scored 18.3% and 16.0% against 30.8% for a symbolic pipeline in Compositional Neuro-Symbolic Reasoning. Neither result is about layouts. The planner-swap comparison, stratified by constraint count, settles the question directly. If the LLM planner matches us at six or more interacting constraints, the planning claim fails.
Why not just iterate with a VLM critic? It is strong, as its reported gains show. Its weak points are a judge that can hallucinate and edits that can undo constraints that were already satisfied. Our first ablation tests exactly this by swapping only the verifier. We will also measure how often a constraint that held at round $t$ breaks at round $t+1$ under each method.
Why not just let an LLM translate free English into ASP and skip the controlled language? This is a real threat to the accessibility half of the proposal. If the third ablation matches the CNL condition on satisfaction rate, the solver is the contribution and the CNL is optional. In that case we would reframe the CNL as a readable display of the program for checking, not as an input language. The user study described below decides which framing holds.
Why not just use attention heuristics or sliders? Attend-and-Excite states that it cannot handle complex spatial relations or negation. Sliders such as CompSlider adjust continuous intensities, but even after reducing entanglement to 14.04% they cannot express counting or exclusion rules. We include the heuristics as baselines rather than assume they lose.
Proposed Experiments
The evaluation runs on four benchmarks. From T2I-CompBench++ we use the numeracy, 2D/3D spatial, and complex-composition splits. We also use ConceptMix at $k=7$. The third is a new suite of 600 constraint-heavy requests written in parallel English and CNL, covering negation, exact counts, adjacency exclusion, and conditional placement, stratified by two to eight interacting rules. The fourth is 150 deliberately unsatisfiable requests.
The primary metric is the all-constraints-satisfied rate. A detector-based checker will score this, and because our own verifier uses detections, the same detector would favor our method. To guard against that, a human audit of 500 images per method provides the decisive score, and we report detector–human agreement. Secondary metrics are B-VQA, CLIP-IQA for image quality, wall-clock time per accepted image, and, on the unsatisfiable set, detection rate and whether users judge the returned conflict explanation correct.
The decisive checks are three. The first is whether our advantage over both main baselines widens with constraint count while staying flat on single-relation prompts. The second is whether the exact verifier beats the VLM critic when planning is held fixed. The third is whether, in a study with roughly 20 non-logician participants (designers and, where we can recruit them, domain experts such as facility planners), people write correct CNL specifications faster and with fewer errors than they write equivalent ASP. A further question for that study is whether participants catch translation errors in LLM-produced programs more reliably when those programs are shown back to them as CNL.
References
- Continuous Diffusion Models Can Obey Formal Syntax
- Constraints-Guided Diffusion Reasoner for Neuro-Symbolic Learning
- Compositional Neuro-Symbolic Reasoning
- MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation
- Enhancing Image Layout Control with Loss-Guided Diffusion Models
- Diffusion Self-Guidance for Controllable Image Generation
- A Tool for Encoding Controlled Natural Language Specifications as ASP Rules
- CNL2ASP: Converting Controlled Natural Language Sentences into ASP
- When Prompt Engineering Meets Software Engineering: CNL-P as Natural and Robust "APIs'' for Human-AI Interaction
- Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis
- T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation
- Iterative Refinement Improves Compositional Image Generation
- Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models
- CompSlider: Compositional Slider for Disentangled Multiple-Attribute Image Generation