TL;DR
Researchers have developed a multimodal machine learning pipeline that reconstructs lost Traditional Chinese Gardens from historical text descriptions. By fine-tuning a DALL·E model on a custom dataset of garden layouts and text, the system generates 2D semantic maps that are then automatically converted into 3D models, offering a scalable way to restore architectural heritage.
What Problem Does This Paper Address?
Throughout history, more than half of the traditional gardens in cities like Suzhou have been demolished. While rich textual documentation of these gardens exists, visual records are often non-existent.
Restoring these gardens currently relies on "speculative design"—architects manually interpreting ancient texts to create analogical drawings. This process is incredibly slow (often taking over a year per garden), requires deep domain expertise, and doesn't scale. The authors aim to bridge this gap by using AI to translate "design intent" (text) into "design outcome" (3D space) at scale.
Methods
The authors propose an end-to-end pipeline consisting of three main stages:
- Multimodal Dataset Creation: They translated historical literature into a standardized format. Instead of raw prose, they used "Pairwise Spatial Relationship Descriptions" focusing on cardinal directions (North, Southeast, etc.) and distance (Far vs. Near).
- Text-to-Image Generation: They utilized the DALL·E architecture, which consists of two parts:
- A discrete Variational Autoencoder (dVAE) to compress garden images into compact visual tokens.
- An Autoregressive Transformer to learn the joint distribution between text tokens and these visual tokens.
- 3D Parametric Reconstruction: The output of the AI is a 2D semantic segmentation map. This map is fed into Grasshopper (a visual programming tool for Rhino 3D). Using a tool called "Rooster," the system extracts vectors from the colors and procedurally extrudes them into 3D architectural elements like pavilions, corridors, and rockeries.
Datasets
The researchers created the first-ever dataset specifically for Traditional Chinese Garden design:
- Source: 40 gardens from the Collection of Traditional Chinese Garden from Each Dynasty.
- Processing: Original CAD drawings were cropped into 256x256 sub-images, resulting in 41–96 pieces per garden.
- Augmentation: To overcome data scarcity, they used rotation and flipping to expand the dataset eight-fold, resulting in 2,284 text-image pairs.
- Representations: They tested six different data groups, comparing simple line drawings against color-coded semantic segmentation maps.
Key Insights and Ideas
- Semantic Maps > Line Drawings: One of the most important findings was that the model converged much faster and more accurately when trained on color-coded semantic maps (where each color represents a specific element like "water" or "pavilion") rather than traditional line drawings. The high contrast and clear labeling helped the AI distinguish between complex architectural features.
- Standardizing "Architectural Language": The authors found that natural language is often too vague for precise reconstruction. By canonicalizing text into structured spatial relationships (e.g., "Pavilion is North of the Pond"), they provided the model with a clearer "logic" of garden design.
- Closing the Loop: Most text-to-image research stops at the 2D image. By adding the Grasshopper post-processing step, the authors turned a generative art project into a functional architectural tool.
Key Results
- Convergence: Models trained on color segmentation (Groups D, E, F) showed significantly lower loss and better fidelity than those trained on line drawings (Groups A, B, C).
- Optimal Configuration: Through CLIP-based evaluation, the researchers found that the model performed best when using color-encoded images paired with text that directly described both orientation and proximity.
- Speed: While manual restoration can take a year, this pipeline generates 2D layouts and 3D references almost instantaneously once trained.
Limitations and Future Work
- Geographical Bias: The current dataset is heavily focused on gardens from the Pan-Yangtze River delta. The authors acknowledge that garden styles vary across China and need more diverse data.
- Granularity: The current labels are somewhat broad. Future iterations could include more specific building types or more detailed botanical information.
- Human-in-the-Loop: The authors emphasize that this is a tool for architects, not a replacement. The "bias" of the human designers who curated the dataset is present in the AI's output, and human judgment is still required to refine the final 3D restorations.
Summary updated