Some image editors rebuild a rough 3D model of the picture and use it as a handle. Image Sculpting fits a mesh, Generative Blocks World fits a set of convex blocks, and BlenderFusion lifts objects into a graphics engine. These editors move objects far more faithfully than editors that work only on pixels. Their weakness is that a person still has to drag the 3D pieces by hand, and the pieces cannot change topology: blocks cannot gain a hole, and a CAD program's numbers cannot grow a new handle. We propose to write the 3D scaffold as a short program in a constructive-solid-geometry language, meaning shapes built by adding and subtracting simple solids. A small language model trained from scratch on millions of such programs, and on edits to them, would turn instructions like "drill a hole through the tabletop" or "make the mug's handle thicker" into program edits. The edited program is then rendered to depth plus warped texture and handed to a depth-conditioned image generator.
The bet comes from domain-specific language modeling. MonoCoder showed that a 0.9B model trained only on high-performance-computing C code, with variable names and literals stripped so that the model must learn structure, beats larger general models on that code. We apply the same recipe to a narrow, structured "domain language": the programs that describe image scaffolds. If the recipe transfers, a compact specialist should produce scaffold edits that are more valid, more precise, and more often topology-correct than a prompted frontier LLM or a LoRA-tuned general code model. The scaffold would also give image edits geometric accuracy that 2D instruction editors lack.
Core Claim
Treating the 3D scaffold as a program and pre-training a language model on that program language does two things existing scaffold editors cannot. It makes discrete topological edits (adding or removing holes, parts, or cavities) reachable from text. It does this without hand manipulation, while keeping the geometric fidelity that scaffold-based editors have over 2D editors. The specific ingredient under test is structure-only pre-training on scaffold programs and their edit pairs. We claim this beats general-model prompting and parameter-efficient tuning on equal data. If it does not, the paper's claim reduces to "programmatic scaffolds help," and we would report it that way.
Where This Sits in Prior Work
Three scaffold editors supply our rendering pipeline and define the limits we target. Image Sculpting de-renders an image into a mesh, edits it with graphics tools, and re-renders through a depth-conditioned diffusion pipeline. Generative Blocks World decomposes scenes into convex polytopes and warps source pixels into a "texture hint" for a depth-conditioned flow model. BlenderFusion composites objects lifted into Blender. All three are driven by manual 3D manipulation. Blocks World's convex parts also fail on concave or holed shapes. Our scaffold adds subtraction to blocks, so concavity becomes expressible, and replaces the hand with a language model.
Two works bound the problem from the topology side. Differentiable 3D CAD Programs for Bidirectional Editing compiles a CAD language into a differentiable graph so that vertex drags update continuous parameters. It has no language interface and explicitly cannot make discrete structural changes. We keep its program-as-scaffold view and move the discrete part to a learned model. 3D Mesh Editing using Masked LRMs achieves genus-changing edits by masked inpainting in a learned 3D representation. It lacks the explicit pixel warping that Blocks World uses to preserve fine texture. Our renderer uses texture hints wherever the program edit leaves surfaces intact and masked generation only where the edit creates new surface, which addresses this tension directly.
On the language side, MonoCoder supplies the method: in-domain pre-training with local-semantics elimination, which anonymizes names and literals after parsing to syntax trees. A Comparative Study of PEFT Methods for Python Code Generation motivates the choice by reporting that domain pre-training predicts downstream performance more strongly than the choice of fine-tuning method. That evidence comes from Python, not geometry, so we test it rather than assume it. Model2Scene aligns CAD geometry with static word embeddings for discriminative segmentation. It neither generates nor edits, and it does not model code. The work closest to our idea is the CAD-program paper above. The new elements here are the learned language model over scaffold programs and the coupling to image re-synthesis.
Research Program
Scaffold language. We define a small constructive-solid-geometry language whose primitives are convex polytopes, cylinders, and boxes. Its operators are union, subtraction, transform, and repeat. Every program compiles to a signed-distance field and a mesh.
Corpus. We fit programs to Objaverse and ABO meshes through greedy convex decomposition followed by residual subtraction search. We then generate edit pairs automatically by applying random program mutations: inserting a subtraction (a hole), deleting a part, duplicating with repeat, or rescaling a subtree. Each mutation yields a (source program, target program) pair whose ground-truth geometry is exact.
Pre-training. Following MonoCoder, we parse programs to syntax trees and anonymize part names. We also quantize numeric literals into relative bins, so the model learns structure rather than memorized constants. A roughly 1B decoder is trained on programs and edit diffs.
Language grounding. A small supervised set of instruction-to-diff pairs is built by captioning rendered before/after mutation pairs with a vision-language model and then human-filtering a subset. This set is shared by all language-model arms.
Image re-synthesis. We fit a program to the input image's object using monocular depth and segmentation, as in BlenderFusion. We apply the predicted diff and render depth. Unchanged surfaces receive texture hints warped through point correspondences, as in Generative Blocks World. Surfaces created by the edit are masked for generation. A depth-conditioned generator produces the output image.
Baselines
The naive reference is returning the input image unchanged, together with an instruction-tuned 2D editor with no scaffold. For the 2D editor we use an SD3 model fine-tuned on MultiEdit. On MultiEdit-Test, MultiEdit reports that its fine-tuned SD3 raises DINO from 0.5751 to 0.7359 and that ME-UEdit-DMTL reaches CLIPimg 0.8174. These numbers show what pure 2D instruction editing achieves without geometry. Our method must match their DINO on MultiEdit-Test's compatible categories while beating them on depth error.
The strongest scaffold competitors are run with oracle edits. For continuous moves, this is Generative Blocks World with the target block transforms supplied by hand. It reports PSNR 18.7 and SSIM 0.874 on its camera-move benchmark, and it serves as an upper bound that our text-driven system should approach within a stated margin, about 1 dB PSNR. For multi-object scenes, BlenderFusion reports PSNR 18.90 and FID 9.11 on MOVi-E. For topology, Masked LRM is driven by an oracle edited view. It reports a 2.56 PSNR improvement on ABO, but no absolute number was found on Lacuna. All systems use the same input images, the same depth estimator, and the same generator backbone where architectures permit.
The strongest language competitors receive the identical instruction set, the language specification, and the same few-shot examples. The first is a prompted frontier LLM (GPT-4o) writing diffs. The second is a general code LLM of comparable size, tuned with LoRA on exactly our program corpus and supervised set under a matched token and compute budget. No reported result was found on Lacuna for any LLM generating scaffold-program edits.
Two ablations isolate the key ingredients. The first removes anonymization from pre-training. The second removes pre-training entirely, using supervised data only. Beating the language baselines means higher diff validity and higher post-edit volumetric IoU, with topology (genus) correct on the held-out mutation types.
Why not just…?
Why not just prompt a frontier LLM to edit the program? This is the real threat. General models write OpenSCAD-like code reasonably well, and our language is small. The evidence that specialists win comes from other code domains: MonoCoder on HPC code and the PEFT study on Python. That evidence is suggestive but not decisive for geometry. The language-model comparison settles the question. If GPT-4o with the specification matches us on validity and IoU, we drop the pre-training claim and report the system as a scaffold-program editor with an off-the-shelf model.
Why not just LoRA-tune a general code model on the same programs? This competitor isolates whether pre-training from scratch matters or whether the data alone does the work. It is run on identical data and compute. We expect the specialist to win mainly on numeric precision and on rare topological operators, where anonymized structural training should matter most. A tie would be an informative negative result.
Why not just edit in 2D with an instruction editor? Generative Blocks World reports that a 2D-guided method, LooseControl, reaches only PSNR 6.65 and SSIM 0.670 under camera moves, compared with 18.7 and 0.874 for the block scaffold. BlenderFusion's scaffold beats engine-free baselines on MOVi-E by 18.90 versus 14.06 PSNR. We expect a similar gap in depth error on topological edits, and the comparison on our new test set measures it directly.
Why not just use Masked LRM for topology? It handles genus changes well but needs an edited conditioning view and loses fine texture. The texture-preservation comparison below tests whether our hybrid of warping and masking keeps the hint's advantage.
Proposed Experiments
The decisive test set is ABO-TopoEdit. We hold out real ABO objects, apply held-out mutation types (programs and meshes), render ground-truth before/after images, and pair each with a human-written instruction. This gives exact targets for topology edits, which existing benchmarks lack. Program-level metrics are diff validity, volumetric IoU, Chamfer distance, and genus accuracy. Image-level metrics are PSNR, SSIM, LPIPS, DINO, and depth RMSE.
We also report SculptingBench depth RMSE, DINO, and CLIP. No numeric values were found on Lacuna for that benchmark, so we rerun Image Sculpting. We report the Blocks World camera-move test and MOVi-E to show that the text interface costs little relative to manual oracles.
The central checks are two. First, does the specialist beat the prompted LLM and the LoRA baseline on genus accuracy for held-out mutations? Second, does texture warping on preserved surfaces beat full masked generation on LPIPS inside unedited regions? A failure on the first check reframes the contribution. A failure on the second reverses our design choice in favor of Masked LRM-style re-synthesis.
References
- Image Sculpting: Precise Object Editing with 3D Geometry Control
- Generative Blocks World: Moving Things Around in Pictures
- BlenderFusion: 3D-Grounded Visual Editing and Generative Compositing
- Differentiable 3D CAD Programs for Bidirectional Editing
- 3D Mesh Editing using Masked LRMs
- MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks
- A Comparative Study of PEFT Methods for Python Code Generation
- Model2Scene: Learning 3D Scene Representation via Contrastive Language-CAD Models Pre-training
- MultiEdit: Advancing Instruction-based Image Editing on Diverse and Challenging Tasks