Speeding up Computational Morphogenesis with Online Neural Synthetic Gradients

Yuyu Zhang, Heng Chi, Binghong Chen, Tsz Ling Elaine Tang, Lucia Mirabella, Le Song, Glaucio H. Paulino

Designing the perfect bridge, airplane wing, or engine component isn't just about aesthetics; it’s a grueling mathematical marathon called PDE-constrained optimization. In a new paper, researchers from Georgia Tech and Siemens Corporate Technology introduce a way to sprint through this process using Online Neural Synthetic Gradients (ONSG).

TL;DR

The authors propose a framework that speeds up large-scale 3D structural design by up to 7.5x. By training a neural network during the design process to predict "synthetic" gradients, they can skip expensive physics simulations while maintaining—or even improving—the quality of the final structure.

What Problem Does This Paper Address?

In the world of "Computational Morphogenesis" (more commonly known as topology optimization), engineers try to find the best distribution of material to support a load. This involves solving Partial Differential Equations (PDEs) to understand how a structure responds to stress.

For high-resolution 3D designs with millions of variables, this is incredibly slow. Standard methods require hundreds of iterations, and each iteration requires solving a massive system of linear equations.

Previous attempts to use Deep Learning (DL) tried to predict the final design in one shot (offline learning). However, these "iteration-free" methods have major flaws:

  1. Data Cost: Generating training data requires solving thousands of designs from scratch.
  2. Generalization: A model trained on 2D squares can't design a 3D bridge.
  3. Physical Accuracy: The results often have "floating" parts or disconnected members because the model doesn't truly understand the underlying physics.

Methods

The researchers introduce ONSG (Online Neural Synthetic Gradients), which shifts the paradigm from offline to online learning.

1. Online Learning & Synthetic Gradients

Instead of pre-training, the model starts "tabula rasa" (blank slate). During the first few iterations of the optimization, it collects "exact" gradients calculated via traditional physics solvers. It uses this history to train a neural network. Once the network is "smart" enough, the system switches to using synthetic gradients—cheap predictions from the neural network—to drive the optimization, skipping the expensive physics solves.

2. The Two-Scale Optimization Scheme

To handle millions of variables without blowing up GPU memory, the authors use a "two-scale" trick. They divide the 3D design space into small local blocks (e.g., $5 \times 5 \times 5$ elements).

  • Input: The model looks at the local material density and a "coarse-grained" global physics state.
  • Output: It predicts the gradient for just that local block. Because the input/output size of the model stays the same regardless of how big the total bridge is, the method is infinitely scalable.

Datasets

The authors evaluated their method on two classic structural engineering benchmarks, scaled up to massive 3D resolutions:

  • Cantilever Beam: A beam fixed at one end with a load at the other.
  • MBB Beam: A beam supported at both ends with a central load.
  • Scale: They tested meshes ranging from 86,000 to 1.5 million design variables, representing some of the largest problems tackled by DL-augmented optimization to date.

Key Insights and Ideas

  • The "Shortcut" Strategy: You don't need to solve the physics perfectly at every single step. Synthetic gradients act as a high-speed shortcut that keeps the optimization moving in the right direction between "reality checks" (exact solves).
  • Scalability via Locality: By treating the design as a collection of local blocks rather than one giant image, the model bypasses the "capacity limit" of deep learning. A small, efficient network can design a massive structure.
  • Escaping Local Minima: Interestingly, because synthetic gradients are approximations, they introduce a tiny bit of "noise" that can actually help the optimizer jump out of poor local minima, sometimes resulting in better final designs than the standard "perfect" solver.

Key Results

  • Massive Speedups: On a 3D problem with 1.4 million variables, ONSG achieved a 7.5x speedup compared to a state-of-the-art GPU-accelerated solver (PCG with Jacobi preconditioner).
  • High Fidelity: Unlike previous DL methods that produced "blurry" or broken designs, ONSG produced crisp, physically sound structures that were visually indistinguishable from (or superior to) standard results.
  • Memory Efficiency: The two-scale approach used only ~0.9GB of GPU memory for a problem where a standard "single-scale" DL model would have required over 3GB (or failed entirely).

Limitations and Future Work

While impressive, the method still has some hurdles:

  • Hyperparameter Tuning: The "block size" ($N_B$) and the frequency of "reality checks" ($N_F$) are crucial. If the block size is too large, the model becomes too heavy; if it's too small, it loses global context.
  • Initial Overhead: The first few iterations still require exact solves to "warm up" the model.
  • Broadening Scope: The authors focused on structural stiffness (compliance). Future work is needed to see if this applies to more complex physics, like fluid flow or heat transfer, where the gradients might be less "predictable."

Paper Citation: Zhang, Y., Chi, H., Chen, B., et al. "Speeding up Computational Morphogenesis with Online Neural Synthetic Gradients." arXiv:2104.12282v2 [cs.AI].

Summary updated

Abstract

A wide range of modern science and engineering applications are formulated as optimization problems with a system of partial differential equations (PDEs) as constraints. These PDE-constrained optimization problems are typically solved in a standard discretize-then-optimize approach. In many industry applications that require high-resolution solutions, the discretized constraints can easily have millions or even billions of variables, making it very slow for the standard iterative optimizer to solve the exact gradients. In this work, we propose a general framework to speed up PDE-constrained optimization using online neural synthetic gradients (ONSG) with a novel two-scale optimization scheme. We successfully apply our ONSG framework to computational morphogenesis, a representative and challenging class of PDE-constrained optimization problems. Extensive experiments have demonstrated that our method can significantly speed up computational morphogenesis (also known as topology optimization), and meanwhile maintain the quality of final solution compared to the standard optimizer. On a large-scale 3D optimal design problem with around 1,400,000 design variables, our method achieves up to 7.5x speedup while producing optimized designs with comparable objectives.
BibTeX
@inproceedings{zhang2021speeding,
  author = {Yuyu Zhang and Heng Chi and Binghong Chen and Tsz Ling Elaine Tang and Lucia Mirabella and Le Song and Glaucio H. Paulino},
  title = {{Speeding up Computational Morphogenesis with Online Neural Synthetic Gradients}},
  booktitle = {IJCNN},
  year = {2021},
  url = {https://openreview.net/forum?id=BMbVcsimZc},
}

Loading related content...

Figure