Telling a CAD Program Which Line Is Wrong: Residual-Attributed Repair for Mesh-to-CAD

Engineers often need to turn a 3D scan or triangle mesh into an editable CAD model, meaning a short program of sketches, extrusions, cuts and fillets that a CAD kernel (the geometry engine inside CAD software) can execute and a designer can modify. Neural models can now write such programs directly from point clouds in under a second, but they write them in one shot. If the program is slightly wrong, for example a missing hole, an extrusion that is 3 mm too short, or a cut that crashes the kernel, nothing in the system notices or fixes it. Methods that do iterate with the kernel in the loop are slow and do not learn. They search over parameters by brute force and take minutes per part.

We propose a middle path borrowed from iterative query refinement in retrieval, where a system looks at the result of a first attempt, finds the specific gap, and issues a targeted follow-up query. Our system executes the draft program, compares the resulting solid with the input geometry, and groups the disagreement into regions of missing and extra material. It then traces each region back to the line of code whose operation produced the nearby faces. The model receives a short numeric diagnosis such as "missing material of roughly 40 mm³ next to faces created by line 7 (extrude of sketch 2)". It answers with a local edit to that line or a new operation inserted after it. A cheap numeric fitter then tunes the edited parameters. The bet is that knowing where the program is wrong, and not just how wrong it is overall, turns kernel feedback into fast, learnable repair.

Core Claim

The claim is that attributing 3D residuals to program lines, using the kernel's record of which operation created which face, is the ingredient that lets a learned editor repair point-cloud-to-CAD programs. It should beat two cheaper options at the same number of kernel executions. The first is drawing more samples and keeping the best. The second is feeding the model only a global distance score or a raw error message. If attribution works, a small model trained this way should close much of the accuracy gap to search-heavy hybrid optimization on hard shapes at a fraction of the wall-clock time. It should also tolerate noisy scans, where residual-based Boolean search breaks down.

Where This Sits

The closest prior work falls into three groups. The first group is one-shot geometry-to-code models. CAD-Recode maps a point cloud to a CadQuery Python script with a Qwen2-1.5B decoder and a small point-cloud projector. PartCAD predicts sketch-extrude parameters from part-aware latents. MiCADangelo decodes constrained cross-section loops and fits extrusion lengths by gradient descent. These supply our draft generator and our input encoder. None of them looks at its own output after execution.

The second group is geometry-to-CAD methods that do iterate. These are the prior work we must differ from most clearly. CADFit puts the OpenCascade kernel in the loop. It computes under- and over-reconstructed volume against a watertight mesh and greedily adds Boolean union and cut operations, found by parameter sweeps, ranked by a learned sketch prior, and pruned afterwards. It reaches a 0% invalid rate, but it needs minutes per part, has no learned editing policy, and fails on non-watertight scans. PS-CAD computes a difference point cloud between the input and the current model. It proposes the next extrusion from plane prompts, but it only ever appends steps, never compiles or repairs earlier ones, and cannot revise a wrong earlier step. Our system differs from both in three ways. It edits existing lines as well as appending new ones. It learns the editing policy with execution rewards. Its residual is stated in terms of program lines rather than raw space.

The third group is text-to-CAD agents with feedback loops. These supply the method template, and they motivate the design choice of numeric over visual critique. RA-CAD trains a Generate–Execute–Critique–Rewrite loop with GRPO, a reinforcement-learning method that scores a group of sampled answers against each other. Its ablations show that the learned critique matters. However, it accepts only text, and its critiques read kernel error strings, not geometry. SPADA repairs code against deterministic geometric assertions. It reports that models self-correct better from numeric assertions than from fuzzy vision-language judgments. Seek-CAD uses step-wise renders critiqued by a vision-language model, and its authors note that the critiques are biased and often wrong. That is why our diagnosis is numeric text, not images. CAD-Coder uses a Chamfer-distance reward during training but is open-loop at test time.

On the query-refinement side, CoV-RAG finds that revising after a draft beats rewriting before retrieval, which is the logic of draft-then-repair. KBQA-R1 shows that multi-turn reinforcement learning over executable atomic actions needs a cold-start fix, namely referenced rejection sampling. We adopt the same remedy below.

Research Program

The system has four parts.

The first part is the draft generator. It is a CAD-Recode-style point-cloud-to-CadQuery model, retrained on the same data so that we control it.

The second part is the attribution engine. It executes the program one operation at a time in OpenCascade and tags every resulting face with the line that created it, in the spirit of Seek-CAD's topological labels. It then computes two point sets with a distance tolerance. Missing points are input points far from the reconstruction. Extra points are reconstruction points far from the input. Each set is clustered, and each cluster is summarized in a few lines of text. The summary gives sign, approximate volume, centroid, bounding box, a dominant fitted plane or cylinder axis, and the program lines whose faces border the cluster. Kernel exceptions are reported against their line in the same format. Because the residual is computed on points with a tolerance, it does not require a watertight input. This is the property CADFit lacks.

The third part is the editor. It is the same language model fine-tuned to emit one structured edit per turn: modify line k, insert after line k, or delete line k. Its context is the current program plus the diagnosis.

The fourth part is the inner numeric fitter. After each structural edit, a few iterations of derivative-free search adjust only the numeric literals on the edited lines to lower a one-sided Chamfer distance. The language model therefore handles topology and intent, and the fitter handles millimetres.

Training proceeds in two stages. The first is supervised repair on synthetic corruptions of ground-truth programs: deleted operations, perturbed parameters, swapped union and cut, wrong sketch plane. The diagnosis is computed from the executed corrupted program, and the target is the inverse edit. This gives millions of free trajectories and solves the cold-start problem, much as KBQA-R1's referenced rejection sampling does. The second stage is GRPO on real draft outputs, rewarded by the gain in volumetric IoU (intersection over union) per edit, with a penalty for invalid programs and for program length. Episodes stop when the residual falls below tolerance or a budget of kernel calls runs out.

The operation vocabulary is extrude, revolve, fillet and chamfer. This matches CADFit's vocabulary so that comparisons are fair. Lofts and sweeps are out of scope.

Baselines

The naive reference point is the one-shot draft generator with no repair. It shows how much any loop adds. On the shared DeepCAD benchmark the file gives no exact CAD-Recode scalars, only "ten-fold improvement in mean CD" and "over 10%" IoU over prior work. We therefore report our retrained copy's numbers directly. For the strongest one-shot published reference on DeepCAD, PartCAD reports Chamfer distance 0.238 and invalid rate 0.91%, and MiCADangelo reports median Chamfer distance 0.20. A repaired system must at least match these figures on DeepCAD to be credible.

The strongest existing competitor is CADFit, because it is the only geometry-to-CAD method with the kernel in the loop and full validity. The file reports that on ABC Hard it "more than doubled the median IoU compared to CAD-Recode", with a 0% invalid rate. No reported absolute IoU value for CADFit on ABC Hard was found on Lacuna, so we will rerun it. We run CADFit on the same stratified ABC Easy/Medium/Hard splits and on the same hardware. We report median IoU against wall-clock time and against kernel calls. Beating CADFit means one of two things: matching its median IoU on ABC Medium and Hard within a few points at one tenth of its time or less, or exceeding it outright on noisy CC3D scans, where its watertight requirement should hurt. A third fair comparison seeds CADFit with our draft, to separate the value of a good initialization from the value of learned edits.

PS-CAD is the second geometry-side iterative baseline. It is evaluated on DeepCAD, where its 10% Chamfer and Hausdorff reductions and 15% edge Chamfer reduction are reported relative to prior work.

Three ablations isolate the key ingredient. Each runs the same editor and fitter with the same kernel budget, but changes what the editor sees. The first ablation shows only the global Chamfer distance and IoU. The second shows the full residual clusters with no line attribution. The third shows kernel error strings only, which is RA-CAD-style feedback transplanted to point-cloud input. A further ablation removes the numeric fitter. Another replaces the trained editor with the same base model prompted zero-shot, which tests the feedback-utilization gap reported for un-fine-tuned models.

Why not just…?

Why not just sample N drafts and keep the one with the lowest Chamfer distance? This is the most serious threat. Independent sampling is cheap, it parallelizes, and it often fixes invalid programs. We give best-of-N exactly the same kernel-execution budget as the repair loop, with N from 4 to 64. Our hypothesis is that sampling plateaus on hard shapes, because the samples share the generator's blind spots: if the model never "sees" a small pocket, none of the 64 drafts contain it. A localized diagnosis names the pocket. The experiment settles this directly. If best-of-N matches repair at equal budget on ABC Hard, the attribution idea fails and we will report that. We also test a hybrid that repairs the best of a few samples.

Why not just optimize the numeric parameters of the one-shot program? Continuous fitting, as in MiCADangelo's differentiable extrusion fitting, fixes dimensions well. It cannot add a missing hole, remove a spurious cut, or recover from a kernel crash. Rather than compete with it, we include it as the inner fitter. The no-editor ablation, which runs the fitter alone on the draft, measures how much of the gain is purely numeric. We expect it to account for most of the improvement on ABC Easy and little on Hard.

Why not just run CADFit, or CADFit seeded with the neural draft? CADFit already reaches full validity. Its costs are minutes per part, greedy append-only Booleans that tend to pile up redundant operations before pruning, and failure on non-watertight scans. Seeded CADFit is included as a baseline. If it matches our accuracy in comparable time, then the learned editor adds nothing beyond a good starting point, and the contribution reduces to initialization.

Why not just give the model rendered images of the mismatch? Seek-CAD's authors report that vision-language critiques of CAD renders are biased. SPADA reports better self-correction from numeric assertions. We add one image-diagnosis ablation at matched token budget to confirm this in the reverse-engineering setting rather than assume it.

Proposed Experiments

The experiments run on DeepCAD, Fusion 360 Gallery, the ABC Easy/Medium/Hard stratification used by CADFit, and CC3D real scans. On CC3D, CAD-Recode reports 89% lower median Chamfer distance and 30% higher IoU than CAD-SIGNet, and no absolute figure is reported, so we rerun it. Metrics are median IoU, mean and median Chamfer distance, invalid rate, program length, wall-clock time per part, and kernel calls per part. Every iterative method is plotted as accuracy against kernel calls.

Three checks decide the outcome. The first is whether full attribution beats both the no-attribution and global-score ablations at equal budget on ABC Hard. This tests the core claim. The second is whether repair beats best-of-N at equal budget on ABC Hard. This tests the main threat. The third is whether, on CC3D, repair holds its gains while CADFit's residual search degrades. This tests the robustness argument.

A secondary analysis checks whether attributed edits land on the line that synthetic corruption actually damaged. That line accuracy is measurable from the supervised data and tells us whether the editor uses the attribution or ignores it. If the first or second check fails, the proposal reduces to "sampling plus parameter fitting is enough", which is itself worth publishing for this field.