SurfScaff3D: Bridging Geometric and Semantic Representations for LLM-based 3D Object Generation

Anay Loya, Rishi Gupta, Arnav Samal, Anirban ChakrabortyarXiv 2026 · arXiv

TL;DR

SurfScaff3D is a training-free framework that enables Large Language Models (LLMs) to generate and edit high-quality, complex 3D objects from text or images. By introducing the Hierarchical Patch Scaffold (HPS)—a representation that combines semantic part-level trees with flexible B-spline patches—the authors bridge the gap between human-like reasoning and precise 3D geometry.

What Problem Does This Paper Address?

Generating 3D content usually requires specialized expertise or expensive, data-hungry AI models. While LLMs are great at reasoning, they struggle with 3D because standard representations (like meshes or raw point clouds) aren't "LLM-native."

Current LLM-based 3D generation faces a trade-off:

  1. Geometric/Parametric methods require LLMs to predict low-level mathematical parameters (like NURBS control points), which is unintuitive for models trained on text and images.
  2. Procedural/Primitive methods use high-level abstractions (like "put a cube here") that LLMs understand, but they are too restrictive to create organic or highly detailed shapes.

SurfScaff3D aims to provide a representation that is both semantically interpretable for the LLM and geometrically flexible enough for complex, organic objects.

Methods

The core of the paper is the Hierarchical Patch Scaffold (HPS) and the SurfScaff3D agentic pipeline.

Hierarchical Patch Scaffold (HPS)

HPS represents an object as a rooted semantic tree (e.g., airplane -> wing -> flap). Each "leaf" in this tree is a 3D surface patch.

  • Surface-Point Interface: Instead of asking the LLM to predict abstract spline control points, the authors have the LLM write Python code to generate a grid of surface points. These points correspond directly to visible geometry, making the task much more intuitive for the LLM. B-splines are then fitted to these points automatically.
  • Patch-Level Surface Variation: To handle organic textures (like the skin of a pineapple), HPS includes parameters for smoothness and frequency. By modulating these, the system can add controlled "noise" to patches, moving beyond perfectly smooth CAD-like surfaces.
Figure 3. Overview of SurfScaff3D. SurfScaff3D iteratively constructs an HPS representation by programmatically generating surface geometry and verifying its local geometry through rendered views. HPS organizes geometry as a hierarchical semantic tree of B-spline surface patches with explicit patch-level control over surface variation, enabling both structured mechanical objects and organic shapes. The examples illustrate how frequency and amplitude help in modelling diverse objects. A critic evaluates rendered outputs and provides targeted feedback, forming a closed-loop refinement process that progressively improves prompt alignment and geometric fidelity. The critic loop executes for a fixed number of N iterations unless an early stopping criterion is met.

SurfScaff3D Framework

The framework uses a dual-agent closed-loop system:

  1. Generator Agent: Receives a prompt and writes Python code to construct the HPS geometry. It can use a render_views tool to "see" its own work and self-correct.
  2. Critic Agent: A vision-capable LLM (like Gemini 2.5 Pro) that inspects multi-view renderings of the generated object. It provides specific, actionable feedback (e.g., "the wing is too short, move the tip further out") to the Generator.

Datasets

The authors evaluated SurfScaff3D across three diverse datasets to test different types of geometry:

  • ShapeNet-Core: For clean, structural CAD-style objects.
  • Toys4K: For diverse, toy-like assets.
  • OmniObject3D: For complex, real-world organic objects.
  • MARVEL-40M+: Used for sourcing high-quality text prompts.
  • ShapeTalk: Used specifically for evaluating 3D editing capabilities.

Key Insights and Ideas

  • Points over Control Points: The "aha moment" is that LLMs reason better about points on a surface than the mathematical control points defining it. This shift makes code generation for 3D much more reliable.
  • Semantic Labels Enable Local Editing: Because every patch is labeled (e.g., chair/leg/front_left), the LLM can edit specific parts of an object without needing to regenerate the entire thing. This preserves the "identity" of the rest of the object.
  • Training-Free is Viable: By leveraging the code-writing and visual-reasoning strengths of frontier LLMs (like Claude 3.5 Sonnet), the authors achieve state-of-the-art results without any task-specific 3D fine-tuning.

Key Results

SurfScaff3D consistently outperformed both fine-tuned and procedural baselines (like LLaMA-Mesh and Articraft) in both geometric fidelity and prompt alignment.

  • Quantitative: On the OmniObject3D dataset, SurfScaff3D achieved significantly better FPD (Fréchet Point Cloud Distance) and higher Elo ratings from LLM judges compared to previous methods.
  • Robustness: It maintained a 0% compilation failure rate during editing tasks, whereas baselines often failed to produce valid geometry when asked to modify existing assets.
  • Qualitative: As seen in the comparisons, SurfScaff3D produces much more "organic" and detailed results than procedural methods, which often look like collections of simple blocks.
Figure 2. Qualitative Comparison of SurfScaff3D. We provide qualitative examples of SurfScaff3D against similar works, on the left we represent methods which allow greater geometric flexibility but require fine-tuning, in the middle we show methods which benefit from the zero-shot generalizability of modern LLMs but suffer from restrictive geometric priors. On the right we show SurfScaff3D using the open-source Kimi K2.6 [20] and Claude Sonnet 4.6 [4] as generation backbones, respectively. Each row corresponds to a single object prompt, all methods are generated using the identical prompts to ensure a fair qualitative comparison.

Limitations and Future Work

  • Watertightness: The model doesn't explicitly guarantee that patches meet perfectly at the edges, which can sometimes leave tiny visible gaps (non-watertight surfaces).
  • Spatial Reasoning: While improved, LLMs still occasionally struggle with complex 3D orientations and relative positioning of many parts.
  • Future Directions: The authors suggest that human-in-the-loop refinement is a natural extension, where a user acts as the "Critic" to guide personalized refinements. They also see potential in integrating more formal geometric constraints to ensure watertightness.

Summary updated

Abstract

Creating high-quality 3D content has traditionally required specialist expertise and experience with complex tools. The growing ecosystem of AI-assisted creativity demands accessible and intuitive 3D generation systems. Recent advances in Large Language Models (LLMs) offer promising directions, yet extending their generative capabilities to 3D remains fundamentally challenging because existing 3D representations are not naturally compatible with language-based reasoning. Current approaches either depend on expensive fine-tuning or impose restrictive structural priors that sacrifice expressiveness and exclude organic shapes. We argue that the mismatch between LLM-native reasoning and conventional 3D parameterizations is a core bottleneck of LLM-driven 3D generation. To address this, we introduce the Hierarchical Patch Scaffold (HPS), a representation designed around semantically meaningful, interpretable part-level abstractions that are naturally understood by both humans and LLMs, enabling expressive, editable, and open-domain 3D generation without imposing restrictive geometric priors or expensive fine-tuning. Building upon HPS, we introduce SurfScaff3D, an interactive framework for text and image-to-3D generation. HPS inherits the flexibility of B-spline patches, while exposing an interface that leverages open-world capabilities of foundation models to reliably generate 3D. Its hierarchical part-level semantic labels enable targeted, natural-language edits at any level of granularity without re-running generation. Further, HPS offers explicit per-patch control over surface detail, enabling more expressive geometry capable of representing arbitrary objects. Extensive experiments against state-of-the-art LLM-based 3D generation methods demonstrate strong geometric fidelity, semantic alignment, and flexible editing across diverse object categories.
BibTeX
@misc{loya2026surfscaff,
  author = {Anay Loya and Rishi Gupta and Arnav Samal and Anirban Chakraborty},
  title = {{SurfScaff3D: Bridging Geometric and Semantic Representations for LLM-based 3D Object Generation}},
  year = {2026},
  eprint = {2611.11678},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11678},
}
Figure