TL;DR
This paper introduces the Fusion 360 Gallery, a novel dataset of 8,625 human-designed CAD programs and an interactive environment (Fusion 360 Gym) for programmatic 3D model construction. This work tackles the challenge of learning from real-world design sequences, enabling the development of AI agents capable of reconstructing CAD programs from target geometries.
What Problem Does This Paper Address?
Creating complex 3D objects in computer-aided design (CAD) software involves a sequence of operations that define the object's geometry. While this programmatic representation is crucial for design, manufacturing, and simulation, there's a significant lack of large-scale, realistic datasets of these design sequences. Existing datasets often focus on the final 3D geometry (meshes, B-Reps) but miss the underlying construction process. This gap hinders the application of machine learning to tasks like CAD reconstruction (recovering the program from the final shape) and programmatic design generation. The authors aim to bridge this gap by providing a dataset and environment that captures human design intent in a structured, executable format.
Methods
The core of this work lies in two main contributions:
- Fusion 360 Gallery Dataset: This dataset comprises 8,625 CAD programs extracted from the Autodesk Fusion 360 Gallery. The programs are simplified to use only two fundamental operations: "sketch" (creating 2D profiles) and "extrude" (giving depth to sketches), along with Boolean operations (join, cut, intersect). This restricted but expressive language allows for a wide range of designs while keeping the problem tractable for machine learning.
- Fusion 360 Gym Environment: This interactive environment acts as a simulator for the simplified CAD language. It allows an "agent" (a machine learning model) to iteratively perform sketch and extrude operations, receiving feedback on the current geometry's state. This setup frames CAD reconstruction as a Markov Decision Process (MDP), where the agent learns a policy to navigate through the sequence of operations to reach a target geometry.
For the CAD reconstruction task, the authors employ a neurally guided search approach. This involves training a neural network (using message passing networks, MPNs) via imitation learning on the ground-truth construction sequences. At inference time, this trained policy guides a search algorithm (like random rollouts, beam search, or best-first search) to explore possible construction sequences within the Fusion 360 Gym until the target geometry is reconstructed.
Datasets
- Fusion 360 Gallery Reconstruction Dataset: This is the primary dataset introduced in the paper. It contains 8,625 human-designed CAD programs, each represented as a sequence of sketch and extrude operations. The dataset is split into a training set of 6,900 designs and a test set of 1,725 designs.
- Existing CAD Datasets: The paper references other CAD datasets (e.g., ShapeNet, ABC, Thingi10k, SketchGraphs) but highlights their limitations in providing programmatic construction sequences suitable for machine learning.
Key Insights and Ideas
- The Power of Simplicity: By restricting the CAD language to just sketch and extrude, the authors demonstrate that a surprisingly expressive range of complex 3D designs can be created. This highlights the potential for simplified DSLs in machine learning for design tasks.
- Human Design Data is Crucial: Experiments show that models trained on human-designed data significantly outperform those trained on purely synthetic data, emphasizing the importance of capturing real-world design nuances and intent.
- CAD Reconstruction as an MDP: Framing CAD reconstruction as a sequential decision-making problem within an interactive environment (Fusion 360 Gym) opens up avenues for reinforcement learning and advanced search techniques.
- Face Extrusion as a Practical Action Representation: For reconstruction, using "face extrusion" (where a face from the target geometry defines the extrusion profile) proves to be a more tractable and effective action representation for learning-based agents compared to generating sketches from scratch.
Key Results
- Dataset Size: The Fusion 360 Gallery dataset contains 8,625 human-designed CAD programs.
- Reconstruction Performance: The neurally guided search approach, using a GCN agent trained on human data, achieved an exact reconstruction rate of 67.5% on the test set within a budget of 100 environment steps, with an average solve time of under 20 seconds per design.
- Agent Performance: Graph-based neural network agents (GAT, GCN) outperformed simpler MLP agents and random baselines in reconstruction accuracy.
- Data Augmentation: Mixing semi-synthetic data with human data (the "aug" agent) showed an early performance advantage in reconstruction IoU.
- Search Strategy: For the given budget, random rollouts slightly outperformed beam search and best-first search in exact reconstruction, likely due to the limited search steps.
Limitations and Future Work
- Limited Operations: The dataset and environment are currently limited to sketch and extrude operations, excluding more complex features like fillets, chamfers, and sweeps.
- Short Sequences: A significant portion of the dataset consists of short construction sequences, which can be solved by naive approaches, potentially masking the true difficulty of the problem for more complex designs.
- Face Extrusion Limitations: Not all designs can be directly converted to face extrusion sequences due to missing intermediate faces in the target geometry.
- Future Work: The authors suggest exploring more sample-efficient search strategies, leveraging dataset constraints for synthesis, developing representations that capture sequential design context (e.g., using self-attention), and expanding to the full range of CAD modeling operations. They also propose using the dataset for other tasks like program synthesis and generative modeling.
Summary updated