Let the Optimizer Judge, Let the Language Model Restructure: Depth-Robust Physical Layout Repair

When a large language model plans a physical arrangement, such as a cluttered tabletop where a cup sits in a bowl on a tray on a box, its output usually violates physics. Objects pass through each other, float above surfaces, or rest on things that cannot support them. Two fixes are currently in use. The first asks the model to critique and revise its own coordinates. The second hands the layout to a differentiable physics optimizer, meaning a program that nudges object poses along gradients of a penalty for collisions and missing contacts. The first fix inherits a well-documented problem: language models are worse at checking solutions than at producing them, and self-critique can drive accuracy down. The second fix cannot make discrete changes. If the bowl simply does not fit on the tray, no amount of nudging will help. Both fixes also suffer from the fact that errors deep in a chain of "A rests on B rests on C" dependencies compound, which is the same depth-driven collapse measured on abstract logic puzzles.

We propose a strict division of labor. The physics optimizer is the only judge of geometry, and the only thing that ever changes continuous poses. When it cannot reach a valid configuration, it reports which node of the dependency structure is infeasible and by how much. The language model is only asked a discrete question about that node and the objects that depend on it: what should this rest on instead, which object should be dropped or swapped, or in what order should things be placed. It never edits coordinates and never judges whether something collides. We predict that this keeps success rates roughly flat as dependency depth grows. We expect three alternatives to degrade with depth: self-refinement, bounding-box feedback prompting, and blind resampling.

Core Claim

Physical layout errors produced by language-model planners come in two kinds. Continuous errors are small penetrations or gaps that gradient optimization repairs reliably. Structural errors are wrong support assignments, impossible nestings, or bad placement orders, and they require a discrete change. We claim three things. Current pipelines fail because they route both kinds through a single channel. Self-correction collapses in this domain for the same reason it collapses in symbolic planning. Depth-driven decay is largely a property of structural errors propagating down support chains.

The testable prediction is this. Suppose optimizer-issued, node-localized infeasibility reports drive discrete-only repairs by the language model. Then success at dependency depth 5 should stay close to success at depth 1. Methods that let the language model edit or judge geometry should instead show the multiplicative decay expected from compounding. If each of $D$ dependent placements independently succeeds with probability $p$, whole-scene success scales as $p^D$. That is the error-propagation argument used to explain multi-hop decay in The Performance Decay of Multi-Step Reasoning Chains.

Where This Sits

Three lines of work motivate the gap.

On the self-verification limitations of large language models on reasoning and planning tasks shows that GPT-4 self-critique lowers accuracy on Graph Coloring from 16% to 2%. It also shows that the model rejects almost all valid Mystery Blocksworld plans, with a 97.09% false-negative rate. Position: LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks turns this into a design rule: the model proposes, and a sound external checker such as a plan validator decides. Both papers stay in discrete symbolic domains. Our proposal carries the rule into continuous geometry. That forces a question neither paper faces: what the checker should say when the answer is "no valid pose exists near here" rather than "precondition 3 fails."

On depth, DyVal and PRONTOQA-OOD show that accuracy falls faster with sequential depth than with breadth. PRONTOQA-OOD finds failures moving from depth 2 to 5 even when every single rule is mastered. Relational Programming with Foundational Models shows the decay disappears when chaining is offloaded to a logic engine. We reuse DyVal's method of generating tasks with controlled depth, and VIEIRA's principle of keeping the chain out of the model's context. Our chain is a physical support graph rather than a table of facts.

Two papers already connect these directions, and they are the closest prior work.

PhyScene3D builds a hierarchical scene graph with a language model. It places objects via bounding-box anchors and runs signed-distance-field test-time optimization that pushes colliding meshes apart. A signed distance field is a function giving each point's distance to the nearest surface, negative inside. The resulting collision-free states serve as rewards for reinforcement learning. PhyScene3D never feeds the optimizer's failures back to the generator at test time. Its optimizer can only move meshes, so structurally impossible placements stay broken or get pushed into semantically wrong spots.

Text-to-Scene with Large Reasoning Models does the opposite. It computes bounding-box overlaps and asks a reasoning model to output corrected positions and rotations. This is exactly the self-correction pattern that the Stechly et al. results warn against, and it has not been measured for failure rates.

Our proposal differs from both in the same way. The model never outputs coordinates, and the optimizer never makes discrete decisions.

The physics side supplies the method. CPF balances attractive springs, which pull intended contacts together, against repulsive springs, which push penetrating vertices out. It shows that either term alone fails: repulsion alone makes objects float, and attraction alone makes them sink into each other. ContactOpt shows that allowing about 2mm of penetration gives stable gradients. It also warns that gradient descent gets trapped in local minima on irregular geometry. That warning constrains our design, because a failed optimization is not proof of infeasibility.

Research Program

The physical checker minimizes, for each node $i$ of the support graph and its descendants, an energy

$$E(x) = \sum_{(a,b)\in C} \lambda_a\, d_{ab}(x)^2 \;+\; \sum_{a\neq b} \lambda_r \sum_{v\in a} \max(0,\, \epsilon - \phi_b(v;x))^2,$$

The terms are defined as follows. $x$ collects object poses. $C$ is the set of intended support contacts named by the language model. $d_{ab}$ is the gap between the contact regions of objects $a$ and $b$. $\phi_b(v;x)$ is object $b$'s signed distance evaluated at vertex $v$ of object $a$. $\epsilon$ is a small penetration tolerance in the spirit of ContactOpt. $\lambda_a$ and $\lambda_r$ weight attraction and repulsion. We add a stability test: the center of mass must project inside the support region, checked in a rigid-body simulator.

The optimizer runs from several initializations. If the best residual $E^\ast$ stays above a threshold $\tau$, it issues a structured report. The report names the node, the violated term (penetration, floating, or unstable support), the minimum residual in physical units, and which descendants are affected.

The language model receives only this report and a discrete action menu for that subtree. The menu is to reassign support, reorder placement, substitute or drop an object, or split a group. The optimizer then re-solves only the affected subtree, starting from the repaired structure. Sibling subtrees that already passed are frozen. This localization is the ingredient we expect to prevent depth decay, because a repair at depth 4 does not force regenerating depths 1 to 3.

We will build three artifacts.

The first is a depth-controlled scene generator in the style of DyVal. It samples support graphs of depth 1 to 6 and breadth 1 to 4 over mesh assets from MesaTask-10k. Every instance has a known feasible solution and a controllable density of structural traps, such as a bowl too wide for its assigned tray.

The second is the repair loop described above.

The third is a measurement suite for the self-verification question in geometry. It asks the language model whether a given layout is collision-free and stable, and scores its false-positive and false-negative rates against the optimizer-plus-simulator ground truth. This extends the Stechly et al. analysis to continuous physics.

We will also measure the checker's own false-infeasibility rate: how often it declares a subtree infeasible when the generator knows a solution exists. This number bounds how "sound" the checker really is.

As a secondary testbed, we will apply the same split to multi-step hand-object sequences from ARCTIC. At each step, the language model chooses the discrete contact region, such as handle versus lid rim, as the object's articulation changes. A CPF-style hand-object energy then optimizes the grasp. This tests whether the approach survives the state inversions that Nakamura et al. show break language-model state tracking.

Baselines

The naive reference is single-pass generation. The same language model produces the scene graph and coordinates once, with no correction, and is scored directly.

The first strong competitor is optimizer-only projection in the style of PhyScene3D. We take the single-pass layout and run the identical energy and multi-start budget, but never return to the language model. This is the most direct competitor because it already reduces collisions substantially. PhyScene3D reports a scene-wise collision rate of 41.6% on MesaTask, against 81.5% in the human-annotated data. It reports downstream ManiSkill grasp success of 50.4% against 4.6% for baseline MesaTask scenes. We run it with the same meshes, optimizer, and wall-clock budget as our loop. Beating it means lower collision and floating rates on MesaTask OOD and on our generated scenes, together with higher satisfaction of the prompt's stated relations. That last condition matters because pushing a bowl off a tray to remove a collision should count as a failure.

The second competitor is language-model self-refinement with bounding-box feedback, following Text-to-Scene with Large Reasoning Models. The model sees overlap reports and outputs new coordinates, using the same model and the same number of calls as our loop. No reported collision rate was found on Lacuna for this method. The paper reports only a 4.4 placement-quality score and human-preference win rates, so we reimplement it.

The third competitor is best-of-$k$ with a sound checker, the "sampling secret" baseline of On the self-verification limitations of large language models on reasoning and planning tasks. That paper reports 72% on Blocksworld with $k=25$ against 55% for self-critique. The LLM-Modulo loop in Position: LLMs Can't Plan reports 82% within 15 iterations. Here we sample $k$ full scene graphs, optimize each, and keep the first that passes. We match the total optimizer and language-model budget to our loop.

Two ablations isolate the key ingredients. The first is localized random repair: the same localized report triggers a discrete resample of the failing subtree chosen uniformly from the action menu, without the language model reading the report. This separates the value of localization from the value of language-model reasoning over the diagnostic. The second is unlocalized feedback: the language model receives the report but regenerates the whole scene. This isolates freezing of subtrees that have already passed.

Why not just…?

Why not just run the physics optimizer on whatever the language model outputs? For continuous errors this is the right tool, and our loop uses it unchanged. The question is structural errors. ContactOpt documents that gradient optimization gets trapped on irregular geometry. An optimizer can remove a penetration only by moving an object somewhere the prompt did not ask for. Our generated scenes control the density of structural traps. If optimizer-only projection matches our loop even at high trap density, the language-model repair step is unnecessary and we will say so.

Why not just let a reasoning model fix the collisions itself? The evidence from symbolic domains is that self-critique lowers accuracy, from 16% to 2% on Graph Coloring and from 4% to 0% on Mystery Blocksworld. The Stechly et al. results do not show whether the same holds for geometry, where models may have better intuitions. Our verification suite measures this directly. If the model's false-negative rate on "is this layout valid" is low and coordinate self-refinement does not decay with depth, the threat is real and our argument weakens.

Why not just sample many layouts and keep one that passes? This is the most serious alternative. It is the baseline the Stechly et al. results show often explains apparent self-correction gains. Our argument against it is quantitative. If each of $D$ nodes is independently feasible with probability $p$, a full-scene sample passes with probability $p^D$, so the required $k$ grows exponentially with depth. Localized repair resamples only the failing subtree. That is why the localized-random-repair ablation matters. If it matches our full loop, the benefit comes from localization and not from the language model reading diagnostics. That would still be a useful finding, but a different paper. If best-of-$k$ at matched budget matches both, the proposal fails.

Proposed Experiments

The central experiment plots whole-scene success against dependency depth, from 1 to 6, for every method at matched compute. Success requires three things at once: no penetration beyond $\epsilon$, no floating contact, and stability in the simulator. It also requires every support relation stated in the prompt to hold, checked symbolically. We fit the decay rate per method and compare it with the $p^D$ prediction. Our hypothesis is supported if our loop's slope is significantly flatter than that of self-refinement and best-of-$k$. It must also beat optimizer-only projection at high trap density. We also report penetration depth and intersection volume as in CPF, language-model calls, and wall-clock time.

On the public MesaTask OOD benchmark, we compare scene-wise collision rate against PhyScene3D's reported 41.6%, together with ManiSkill grasp success against its 50.4%. We report relation satisfaction alongside, so that removing objects cannot fake a win.

The verification study reports the language model's false-negative and false-positive rates on layout validity, stratified by depth. It also reports the optimizer's false-infeasibility rate against generator ground truth. If the checker's false-infeasibility rate exceeds the language model's own error rate, the premise that "the optimizer should judge" is wrong at this compute budget.

The ARCTIC extension measures per-step contact-region accuracy and final penetration and motion deviation across multi-step articulated sequences. ARCTIC's reported motion deviation of 8.9mm for ArcticNet-LSTM serves as a reference scale. We compare discrete-only language-model choices against letting the model output full hand poses.

The decisive check is the matched-budget comparison at depth 5 with high trap density. Our loop must beat both best-of-$k$ with the same optimizer and localized random repair. If it does, language reasoning over physical diagnostics adds value beyond search. If only localization wins, the contribution narrows to an efficient structured-search result.