Semantic Direct Modeling

Qiang Zou, Shuo LiuarXiv 2025 · DOI · arXiv

Abstract

Traditional Computer-Aided Design (CAD) workflows require engineers to translate high-level design intent into low-level geometric manipulations, a process that is often tedious and cognitively demanding. This paper presents Semantic Direct Modeling (SDM), a novel framework that enables designers to modify 3D models using natural language and speech. By integrating Large Language Models (LLMs) for intent parsing with a context-sensitive generative AI for feature recognition, SDM bridges the gap between human conceptualization and geometric execution. The system achieves a 97.5% accuracy in command interpretation and significantly outperforms existing state-of-the-art methods in feature identification.

1. Introduction

In modern engineering, "direct modeling" represented a significant leap forward by allowing designers to manipulate geometry without the constraints of a rigid history tree. However, the experience remains fundamentally low-level. To modify a simple feature, such as a slot or a pocket, a designer must manually select every constituent face and edge before defining precise transformations. This creates a disconnect: while engineers think in terms of functional features, they are forced to work in terms of raw geometry.

Semantic Direct Modeling (SDM) addresses this by introducing a "Semantic Layer" that interprets natural language commands and automatically maps them to the underlying Boundary Representation (B-rep) of the CAD model. This allows for a more intuitive, intent-driven design process.

2. The SDM Architecture

The SDM system is built upon two primary pipelines that synchronize human intent with geometric data.

2.1 Human-to-Semantics (The Interpretation Engine)

The first pipeline converts spoken or written commands into structured data. Using OpenAI’s Whisper for speech-to-text, the system captures the designer's input. This input is then processed by a Llama LLM, which has been fine-tuned using Chain-of-Thought (CoT) prompting.

Unlike standard prompting, which often fails to capture the nuances of CAD operations, CoT guides the LLM to systematically identify the feature type, extract operation parameters (such as distance, angle, and direction), and output a structured JSON object. This structured intent serves as the blueprint for the subsequent geometric modification.

2.2 Geometry-to-Semantics (Conditional Feature Recognition)

The second pipeline identifies the specific geometric entities (faces) that correspond to the designer's intent. This is achieved through a novel Conditional Feature Generation approach:

  • B-rep Tokenization: To process 3D geometry within a Transformer architecture, B-rep faces are converted into triangular meshes. Tokens are generated based on triangle centers, normals, and neighbor indices, making the 3D model "readable" for the AI.
  • Cross-Attention Fusion: Rather than using a static classifier, SDM uses a cross-attention mechanism to fuse the text embedding from the LLM with the geometric embedding of the model. This makes feature recognition dynamic; the system identifies faces based on the context of the user's command (e.g., identifying a "pocket" specifically because the user asked to move one).
  • Generative Decoding: A Transformer decoder autoregressively generates the IDs of the faces that match the intent, allowing the system to handle features with varying numbers of faces more effectively than traditional discriminative models.

3. Implementation and Results

The system was integrated with Siemens NX via an API Generation Module that translates the identified face IDs and parameters into executable CAD code.

3.1 Datasets and Training

The feature recognition module was trained on two major benchmarks:

  • MFCAD: 15,488 shapes with 16 machining feature types.
  • MFCAD++: 59,655 models including complex, non-planar geometries.

3.2 Performance Metrics

The SDM framework demonstrated significant improvements over baseline methods:

  • Command Parsing: The introduction of CAD-specific CoT prompts increased LLM accuracy from a baseline of 15% to 97.5%.
  • Feature Recognition: On the MFCAD dataset, SDM achieved an Intersection over Union (IoU) of 99.54% and an Exact Match (EM) rate of 99.20%, consistently outperforming the previous state-of-the-art, UV-Net.

4. Discussion and Future Work

While SDM represents a major step toward semantic CAD, certain limitations remain. The current LLM lacks "spatial awareness"—it cannot yet interpret relative spatial commands like "move the hole to the top-left" because it does not process the visual state of the model in real-time. Future iterations aim to incorporate vision-language models to resolve these spatial ambiguities.

Additionally, the authors plan to expand the system's scope beyond design intent to include analysis intent (optimizing for structural integrity) and manufacturing intent (automated CNC tool path generation), creating a truly end-to-end semantic engineering environment.

5. Conclusion

Semantic Direct Modeling shifts the CAD paradigm from manual geometric manipulation to high-level intent communication. By combining the linguistic reasoning of LLMs with context-aware geometric deep learning, SDM reduces cognitive load and streamlines the design process, paving the way for more intelligent and accessible engineering tools.

Summary updated

BibTeX
@misc{zou2025semantic,
  author = {Qiang Zou and Shuo Liu},
  title = {{Semantic Direct Modeling}},
  year = {2025},
  eprint = {2504.13893},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.13893},
}
Figure