GAOT: Generating Articulated Objects Through Text-Guided Diffusion Models

Hao Sun, Lei Fan, Donglin Di, Shaohui Liu2025 · arXiv

Abstract

Recent advancements in text-to-3D generation have primarily focused on static geometry, resulting in digital models that lack functional utility. We introduce GAOT (Generating Articulated Objects via Text), a novel three-phase framework designed to transform natural language prompts into functional 3D models with moving parts. By decomposing the generation process into point cloud synthesis, hypergraph-based part extraction, and diffusion-based joint prediction, GAOT successfully bridges the gap between high-level language and complex mechanical kinematics.

Introduction

The field of generative AI has seen an explosion in models capable of producing 3D meshes or point clouds from simple text descriptions. However, most existing outputs are essentially "digital statues"—static representations that do not reflect the interactive nature of the real world. In reality, the objects we interact with are articulated: refrigerators have swinging doors, desks have sliding drawers, and faucets have rotating handles.

Generating these objects automatically is a significant challenge because it requires a model to simultaneously master three distinct domains:

  1. Global Shape: The overall visual appearance and structure of the object.
  2. Part Segmentation: The identification of independent components that move separately.
  3. Kinematics: The mathematical constraints of connectivity, such as determining if a joint is revolute (rotating) or prismatic (sliding).

While previous models like NAP generate articulated objects unconditionally (from noise) and others like Cage require pre-defined connectivity graphs, GAOT allows for an intuitive, text-driven workflow.

Methodology

GAOT employs a modular three-stage pipeline to transition from a text prompt to a fully articulated 3D model.

Phase 1: Coarse Point Cloud Generation

The process begins by leveraging a fine-tuned version of Point-E, a text-to-point-cloud model. The authors fine-tuned this model on the PartNet-Mobility dataset to generate a 10,000-point cloud from a prompt (e.g., "a water faucet"). In this stage, the colors of the points are used to represent potential moving parts, providing a rough spatial map of the object's components.

Phase 2: Hypergraph Part Extraction

To transform the raw point cloud into discrete, functional parts, GAOT utilizes a Hypergraph Neural Network (HGNN). Traditional graphs are limited to pairwise relationships, but hypergraphs can model high-level interactions among multiple entities simultaneously. This is crucial for articulated objects, which often feature nested dependencies (e.g., a handle attached to a drawer that is itself inside a cabinet). The HGNN refines the point cloud into a "vertex matrix," where each vertex represents a distinct physical part.

Phase 3: Joint Generating Diffusion

The final stage determines how these parts connect and move. The authors employ a Denoising Diffusion Probabilistic Model (DDPM) to predict the connectivity graph. Starting from a matrix of random noise, the DDPM gradually denoises the data into a valid set of joint parameters. This stage defines the mechanical logic of the object, specifying the type and axis of movement for every connection. The resulting graph-based representation is then rendered into a complete, articulated 3D model.

Experimental Setup and Datasets

The model was trained and evaluated using PartNet-Mobility, a large-scale repository of 3D objects annotated with motion parameters. To ensure model stability, the researchers focused on objects with eight or fewer parts.

Because the original dataset contained only simple category labels, the authors performed Prompt Expansion. They expanded generic labels (like "refrigerator") into more descriptive, unique prompts. This allowed the model to learn a more diverse distribution of shapes and better understand the nuances of natural language.

Results and Discussion

GAOT was benchmarked against several baselines, including combinations of Point-E, Shap-E, and NAP.

Quantitative Performance

GAOT demonstrated superior performance in Coverage (COV), a metric measuring how well the generated models represent the diversity of the reference set. On complex prompts, GAOT achieved a COV of 0.4898, nearly doubling the performance of the Point-E and NAP baseline (0.2857).

Qualitative Rationality

Visual analysis indicates that GAOT produces more "rational" objects than its predecessors. While baseline models frequently produced floating parts, disconnected joints, or incomplete geometries, GAOT’s models featured cohesive shapes with joints that adhered to real-world physics. For instance, cabinet doors generated by GAOT were correctly hinged to the frame, rather than being disconnected or misaligned.

Limitations and Future Work

While GAOT represents a significant step forward, several challenges remain:

  • Complexity Constraints: The current framework is limited to objects with 8 or fewer parts, leaving highly complex machinery for future exploration.
  • Physical Realism: Occasional failure cases occur, such as misaligned parts or unrealistic joint motions (e.g., a handle moving in a physically impossible direction).
  • Scaling: Future research will focus on scaling the model to handle more complex hierarchies and improving the precision of joint parameter estimation to ensure even greater physical accuracy.

Conclusion

GAOT successfully demonstrates that the "Text-to-Motion" gap can be bridged by breaking the generation process into a logical sequence: Shape $\rightarrow$ Parts $\rightarrow$ Joints. By using hypergraphs to capture complex dependencies and diffusion models to solve mechanical connectivity, GAOT moves 3D generation closer to creating truly functional digital twins.

Summary updated

Abstract

Articulated object generation has seen increasing advancements, yet existing models often lack the ability to be conditioned on text prompts. To address the significant gap between textual descriptions and 3D articulated object representations, we propose GAOT, a three-phase framework that generates articulated objects from text prompts, leveraging diffusion models and hypergraph learning in a three-step process.
BibTeX
@misc{sun2025gaot,
  author = {Hao Sun and Lei Fan and Donglin Di and Shaohui Liu},
  title = {{GAOT: Generating Articulated Objects Through Text-Guided Diffusion Models}},
  year = {2025},
  eprint = {2512.03566},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.03566},
}
Figure