Representing 3D Geometry as Executable Code

52 concepts38 papers2 proposals

Traditional deep learning approaches to 3D vision and graphics treat shapes as static spatial data structures: voxel grids, point clouds, or unstructured polygon meshes. While these representations capture surface geometry, they discard the underlying structural logic. A mesh of a chair does not know it has four identical legs, nor does it understand that the seat must remain flat. If you want to modify the chair's height, you must manually warp thousands of vertices, often destroying the object's physical plausibility.

To solve this, a rapidly growing body of research shifts the paradigm from predicting geometry to predicting programs. By representing 3D shapes and scenes as executable scripts—written in domain-specific languages or standard parametric CAD frameworks like CadQuery and Blender Python—generative models can synthesize highly complex, editable, and structurally sound 3D assets.

Why Code Works: Leveraging Pre-trained Logic

Why should we use code as an intermediate representation for 3D shapes? The answer lies in how modern neural networks, particularly Large Language Models (LLMs), are trained. LLMs struggle with raw spatial coordinates and precise numerical geometry, but they are exceptionally proficient at writing code due to massive pre-training on software repositories.

By mapping 3D design sequences to structured, Python-like code, frameworks can exploit this pre-existing programming knowledge for geometric reasoning. As shown by Guan et al. in CAD-Coder: Text-to-CAD Generation with Chain-of-Thought and Geometric Reward, using standard parametric programming languages allows generative models to utilize logical reasoning patterns acquired during general code pre-training. Similarly, Li et al. demonstrate in CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generation that treating CAD sequences as high-level programming code bridges the gap between abstract spatial operations and sequential code logic.

This strategy effectively transforms a difficult spatial synthesis problem into a language translation task. For instance, Makatura et al. note in How Can Large Language Models Help Humans in Design and Manufacturing? Part 1: Elements of the LLM-Enabled Computational Design and Manufacturing Pipeline that using domain-specific languages or intermediate text-based formats allows language models to bypass their inherent arithmetic limitations, relying instead on their coding proficiency to build complex physical designs.

Capturing Structural Regularity and Logic

A core advantage of programmatic representations is their ability to enforce high-level geometric regularities like symmetry, repetition, and topological constraints. In a traditional mesh, a row of ten evenly spaced pillars requires defining ten separate sets of coordinates. In a program, it requires a single for loop.

Early neural shape parsers like CSGNet: Neural Shape Parser for Constructive Solid Geometry demonstrated that parsing visual inputs into executable visual programs composed of primitives and boolean operations yields resolution-independent, highly interpretable outputs. Expanding on this, Tian et al. introduced a neuro-symbolic model in Learning to Infer and Execute 3D Shape Programs that automatically discovers structural regularities using control structures like loops. This ensures that generated shapes maintain physical stability and structural consistency.

This programmatic efficiency extends from individual objects to entire environments. In The Scene Language: Representing Scenes with Programs, Words, and Embeddings, Zhang et al. show that representing 3D scenes through programmatic loops and functions captures structural regularity far more accurately and efficiently than traditional scene graphs, allowing for the precise placement of numerous objects via underlying spatial logic.

Parametric Editability and Modularity

When a model outputs a static mesh, that asset is essentially "frozen." Human artists or engineers cannot easily modify its components. Programmatic representations, by contrast, produce inherently modular and legible intermediate representations.

In MeshCoder: LLM-Powered Structured Mesh Code Generation from Point Clouds, Dai et al. demonstrate that translating 3D geometry into executable scripts allows downstream models and humans to interpret and manipulate structural properties through code analysis rather than raw visual data. This ensures that the output remains fully editable within professional software environments, a benefit also highlighted by Huang et al. in BlenderAlchemy: Editing 3D Graphics with Vision-Language Models.

Furthermore, representing shapes as programs enables high-level semantic interactions that are impossible with monolithic geometry. As Zhang et al. show in ShapeCraft: LLM Agents for Structured, Textured and Interactive 3D Modeling, programmatic shapes allow for precise post-generation editing, part manipulation, and even functional animations because the underlying code explicitly defines the object's components and relationships. This flexibility is critical for domain-specific applications, such as generating simulation-ready sewing patterns from 2D images as explored by Bian et al. in ChatGarment: Garment Estimation, Generation and Editing via Large Language Models.

Outsourcing Precision to External Solvers

Neural networks are notoriously bad at precise spatial math and constraint satisfaction. If an LLM is asked to place an object exactly touching another object, it might output coordinates that result in a slight gap or an intersection.

Programmatic representations solve this by acting as an interface to external geometric engines and solvers. Instead of forcing the neural network to calculate exact coordinates, the network generates code that calls high-level operators—such as 2D-to-3D sweeps, lofting, or boolean operations.

As Hu et al. demonstrate in SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code, generating executable code allows language models to utilize external geometric solvers and professional rendering engines, vastly improving the accuracy of complex object placement. This closed-loop execution paradigm is further refined by Kolodiazhnyi et al. in cadrille: Multi-modal CAD Reconstruction with Reinforcement Learning, where a reinforcement learning pipeline uses programmatic feedback from the executed code to iteratively correct and optimize the generated 3D geometry.

This ability to generate functional, simulation-ready code also extends into robotics and physical tool design. For instance, Gao shows in VLMgineer: Vision Language Models as Robotic Toolsmiths that representing physical tools as modular code structures (like URDF files) allows vision-language models to autonomously design complex hardware without manual templates. Similarly, in ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation, Huang et al. leverage vision-language models to generate executable Python code mapping 3D keypoints to scalar costs, bypassing the difficulty neural models face when directly predicting precise numerical spatial parameters.

Key Foundational Work

<scp>Proto-Plasm</scp> : parallel language for adaptive and scalable modelling of biosystems (2008) — contributed 1 concept elements to this direction.
Automatically Transforming Symbolic Shape Descriptions for Use in Sketch Recognition (2004) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_programmatic_shape_representation_2c8c97c3,
  title = {programmatic shape representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/programmatic-shape-representation/rdr_2c8c97c37c0142c19af6bf97c4a140fc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_programmatic_shape_representation_2c8c97c3_v1,
  title = {programmatic shape representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/programmatic-shape-representation/rdr_2c8c97c37c0142c19af6bf97c4a140fc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake