Scan-to-Program Recovery with Implicit Residuals as the Verifier

Large language models can write shape programs, short scripts that build an object from named parts such as "leg," "seat," or "handle." They are poor at the arithmetic those programs need. They misplace coordinates, get rotations wrong, and leave parts floating. The usual fixes supply the missing precision from outside the model. One fix is a constraint solver that computes positions; another is a checker that reports "part A intersects part B by 2 mm." Both fixes assume the object is fully describable by rigid computer-aided design (CAD) primitives. Real objects are not. A scanned mug has a dent, a chair has a sculpted backrest, a tool handle has an organic grip. Any program the model writes will be wrong somewhere, and today nothing tells the model whether that wrongness is harmless detail or a structural mistake, such as a missing part or the wrong kind of joint.

We propose borrowing the central trick of hybrid 3D reconstruction from human-body modeling. There, a parametric template carries the global structure and a learned implicit surface soaks up the remaining detail. An implicit surface is a function that returns, for any point in space, its signed distance to the surface. We apply the same split to program synthesis from scans. The language model writes only the structure of a program. A differentiable optimizer fits its numbers to the scan. A small neural distance-field residual, attached to each part's local coordinate frame and penalized for size, absorbs what the program cannot express. The size, sign and location of that residual become the feedback the model reasons over. A large, spatially coherent residual means "a part is missing or mis-structured here," while a small, diffuse residual means "detail, leave it." Deleting a requirement to make an error message go away is a known pathology of these repair loops. Under this design it makes the measured residual worse, because the scan, not the model's own specification, defines what is true.

Core Claim

A penalized, part-anchored implicit residual is a better verifier for language-model shape programs than either rendered images or discrete geometric assertions. Our claim is that it separates the two kinds of error the model makes: numeric errors, which an optimizer fixes, and structural errors, which only the model can fix. This separation should let an off-the-shelf model recover editable programs for objects with non-CAD detail. We test this with an editability measure that a pure implicit fit cannot game. The new element is using the residual of a hybrid representation as the repair signal in a program-synthesis loop, with an acceptance rule tied to observed geometry.

Where This Sits

The reconstruction side supplies the mechanism. IP-Net fits the SMPL body model (a standard low-dimensional parametric human mesh) to a predicted inner body surface, then adds per-vertex displacements for clothing. Fitting SMPL directly to clothed scans gave 14.85 cm registration error, against 3.67 cm for this two-stage fit. Our program-then-residual split is the analogue, with a language model writing the template instead of a fixed statistical model. AlignSDF shows that evaluating a distance field in a pose-normalized frame, derived from a parametric hand, cuts hand and object shape error by 29.4% and 20.5% on DexYCB. We reuse this by expressing each residual in its part's local frame. DyTact binds deformation surfels (small oriented surface patches) to a template and densifies them only where needed. Its localized residual capacity is the design we follow. These papers also constrain us. They note that everything downstream breaks when the initial parametric fit is wrong. In our setting that is exactly the case the language model must detect and fix.

The language-model side supplies the problem and several negative results. 3D-PreMise finds that GPT-4 with one-shot chain-of-thought (step-by-step prompting) reaches only 19.3% Pass@5 on 57 CAD objects, with spatial precision errors dominant. It also finds that feedback from rendered images suffers from 3D hallucinations. AIDL lets the model state relations while a solver computes coordinates. It also documents the intent-pruning failure, in which the model drops requirements when the solver keeps failing. SPADA shows that deterministic numeric assertions double assembly pass rate over zero-shot, from 20.4% to 41.9%. Its verifiers are discrete and CAD-only, and its task is text-to-CAD with no observation to match. We adopt AIDL's division of labor and SPADA's insistence on quantitative feedback. We differ in that the specification is a scan and the feedback is a continuous, localized residual rather than a pass/fail interference test.

Spatial reasoners such as DiSR and SpatialReasoner serialize perception into text for a model to reason over. DiSR reaches near-perfect accuracy given ground-truth 3D evidence, which locates the bottleneck in perception. Both pipelines are one-way and use boxes rather than surfaces. Our loop runs in both directions: the model's structural choices change the fit, and the fit's residual changes the model's next choice. Finally, Evaluating Intermediate Reasoning of Code-Assisted LLMs shows that a correct final output can hide unsound program logic. This is a warning we take seriously. A flexible residual could produce low Chamfer distance on top of a structurally wrong program. We therefore penalize the residual and measure program structure directly.

Research Program

The system has four components. The first is a relational shape language in the style of AIDL. The model declares parts drawn from analytic primitives (boxes, cylinders, swept profiles, fillets, booleans) and states relations such as "legs attached at seat corners" or "handle centered on body." Every numeric value is left as a free parameter. The second component is a differentiable fitter. Primitive signed distance functions are analytic, so a program with parameters $\theta$ defines a distance field $d_P(x;\theta)$. We fit $\theta$ together with per-part residual networks $r_k$, each evaluated in part $k$'s local frame $T_k(\theta)$, by minimizing

$$ \mathcal{L}(\theta, r) = \sum_{x \in S} \big| d_P(x;\theta) + \textstyle\sum_k w_k(x)\, r_k(T_k(\theta)^{-1}x) \big| + \lambda \sum_k \|r_k\|_{L^1} , $$

where $S$ is the set of scan points (which should have distance zero), $w_k$ is a soft assignment of points to parts, and $\lambda$ sets how much geometry may be explained outside the program. We add an eikonal term and free-space samples in practice. The $L^1$ penalty is what gives the residual its meaning: it pushes the residual to be sparse, so that it concentrates where the program is actually wrong.

The third component is a residual report written in text. It contains, per part, the mean and maximum residual magnitude and its dominant sign (material missing or excess). It lists clusters of scan points that no part explains, with their centroid and extent expressed relative to named parts. It also gives signed penetration depths between parts. Following divide-and-conquer prompting, which shows that isolating subproblems in a short context reduces drift, the report is organized per part rather than as one global summary.

The fourth component is an acceptance rule. The model may add, remove, retype or re-relate parts. An edit is kept only if it lowers $\mathcal{L}$ plus a program-length penalty after refitting. Removing a part leaves its region unexplained and raises the residual, so intent pruning is rejected automatically rather than prohibited by prompting.

Baselines

The naive reference is a single-shot program written by the model from a multi-view rendering and point summary of the scan, with no fitting and no feedback. This is the 3D-PreMise setting, where GPT-4 with one-shot chain-of-thought reports 19.3% Pass@5 on its own 57 objects, according to 3D-PreMise.

The strongest existing method for repairing language-model CAD programs is SPADA's compile-test-repair loop with deterministic geometric assertions (SPADA). It reports 41.9% assembly All Pass Rate and 78.9% Part IoU on SPADA-Bench-Verified. That result is for text-to-CAD. There is no reported result on Lacuna for scan-conditioned program recovery, so we re-run SPADA's loop on our scan tasks. We give it the scan and add a scan-deviation assertion (global Chamfer distance and interference checks) so that it is not disadvantaged by lacking an observation. We also run a visual self-refinement baseline in the 3D-PreMise style, which feeds renders back to a multimodal model.

On the reconstruction side, the strongest competitor for geometric fidelity is a pure neural signed distance field fit to the scan. It will almost certainly win on Chamfer distance and cannot be edited. It is the upper bound on fit and the lower bound on editability.

All language-model systems use the same model, the same budget of repair rounds and tokens, the same scan inputs and the same wall-clock limit for fitting. Prompts for every system are tuned on a shared held-out development split.

Three ablations isolate our key ingredient. The first removes the residual entirely, leaving a program with fitted parameters, which tests whether the residual is needed at all. The second keeps the residual but replaces the per-part report with a single global Chamfer number, which tests whether localized structure is what helps. The third disables the acceptance rule, which tests whether intent pruning returns. A fourth ablation expresses residuals in world coordinates instead of part frames, testing the AlignSDF-style normalization. Beating SPADA means higher structural accuracy and higher edit fidelity at equal or lower Chamfer distance and equal budget. Beating the no-residual ablation by a clear margin on objects with non-CAD detail, and matching it on pure-CAD objects, is the signature we expect if the mechanism works as claimed.

Why not just…?

Why not just fit a neural distance field to the scan and skip the language model? It will reconstruct better, and IP-Net shows why unstructured fits are fragile under sparse input. The main answer is that a pure field has no parts to edit. Our edit-fidelity test changes a named parameter, such as leg length or handle radius, in the recovered program and compares the result against the ground-truth program edited the same way. A pure field cannot take this test, and we report that as a structural limitation rather than hiding it.

Why not just give SPADA-style feedback a Chamfer-distance assertion? This is the real threat. A global error number with interference checks may already tell the model enough, in which case the residual adds nothing. The global-Chamfer ablation settles it directly. If per-part residual reports do not improve structural accuracy over it, particularly on objects with a missing or misplaced part, the central claim fails. We would then report the simpler system as the recommendation.

Why not just let the model compute the numbers with chain-of-thought or visual feedback? 3D-PreMise finds that spatial precision is the dominant failure and that render-based feedback hallucinates. Beyond Code finds that self-generated intermediate steps often hurt because early mistakes compound. Our naive and visual baselines re-measure this on scans. We expect the gap to be largest on multi-part assemblies.

Why not just let the residual absorb everything and trust low Chamfer? Following Evaluating Intermediate Reasoning, a good final output can hide wrong structure. We sweep $\lambda$ and report the fraction of geometry explained by the program alongside structural accuracy. If structure collapses as the residual grows, the design only works in a narrow penalty band, and we will say so.

Proposed Experiments

The main benchmark is built from SPADA-Bench-Verified, whose ground-truth programs give us part structure and editable parameters. We sample point-cloud scans from its assemblies with realistic noise and partial views. We create a "detailed" variant by adding non-parametric deformations: dents, sculpted surfaces, embossing and softened edges, which no program in the language can express. We also convert the 57 objects of 3D-PreMise the same way.

We report four metrics. Structural accuracy is part-count and part-type match against the ground-truth program, plus relation correctness. Geometric fidelity is Chamfer and Hausdorff distance to the clean scan. Edit fidelity is Chamfer distance after a parameter edit, measured against the edited ground truth. Intent retention is the fraction of ground-truth parts present in the final program.

As a secondary probe, we evaluate the parametric-surface split of SURFACEBENCH. There, exact symbolic recovery is 4% for language-model methods, and multi-output coupling is the known weakness. We ask whether "model proposes form, optimizer fits constants, residual localizes misfit" also helps symbolic surface discovery.

Three checks are decisive. First, on the detailed variant, per-part residual feedback should beat global-Chamfer feedback on structural accuracy and edit fidelity, and the no-residual ablation should lose badly on geometric fidelity. On pure-CAD objects the two feedback conditions should be close. Second, with the acceptance rule removed, intent retention should drop measurably across repair rounds, reproducing the AIDL pathology. With the rule in place it should stay flat. Third, in a controlled test we delete one part from a correct program. The residual report should localize the missing region, and the model should restore a part of the right type at a rate well above the global-Chamfer condition. If any of these fails, we learn which part of the claim was wrong rather than only that the system scored lower.