Abstract
Solving Partial Differential Equations (PDEs) is fundamental to engineering but remains computationally intensive for traditional numerical methods. We present the Multi-input and Multi-scale Efficient Transformer (MMET), a novel framework designed to solve PDEs across diverse geometries, boundary conditions, and resolutions. By decoupling mesh encoding from point queries and employing Hilbert curve-based patching, MMET addresses the challenges of multi-input generalization and computational scaling. Our results demonstrate state-of-the-art accuracy and efficiency on complex 2D and 3D engineering benchmarks.
Introduction: The Challenge of PDE Solving
Partial Differential Equations (PDEs) underpin nearly all physical simulations, from aerodynamics to thermodynamics. While traditional methods like the Finite Element Method (FEM) provide high accuracy, their computational cost often prohibits real-time application. Neural Operators have emerged as a faster alternative, yet they frequently struggle with three core issues: generalizing across different physical boundary conditions, handling multi-scale resolutions, and scaling to massive 3D meshes without memory exhaustion.
The MMET Framework
To overcome these hurdles, MMET introduces three primary architectural innovations:
1. Encoder-Decoder Decoupling
Standard Transformers often process inputs as a single sequence. MMET instead treats the geometric and physical mesh as the "source" for the Encoder and the specific coordinates for which a solution is required as "queries" for the Decoder. This decoupling enables the model to perform zero-shot resolution scaling, predicting values at any spatial point regardless of the training resolution.
2. Gated Condition Embedding (GCE)
Handling diverse physical inputs requires distinguishing between actual zero-valued measurements and missing data. The GCE layer utilizes a gating mechanism to resolve this "ambiguity of zero," allowing a single model to process Dirichlet, Neumann, and Robin boundary conditions simultaneously.
3. Hilbert Curve Reserialization & Patching
For large-scale models, MMET maps 2D/3D points into a 1D sequence using a Hilbert curve. This approach preserves spatial locality better than standard scanning methods. These sequences are then "patched" into tokens, drastically reducing the sequence length for the Transformer's attention mechanism while maintaining local geometric fidelity.
Experimental Evaluation
MMET was evaluated on six benchmarks, including Poisson, Darcy Flow, Shape-Net Car (3D aerodynamics), and various heat and stress problems (Heat2d, Beam2d, HeatSink2d).
Key Results
- Superior Accuracy: On the complex
Beam2ddataset, MMET achieved a relative $L_2$ error of $4.64 \times 10^{-2}$ for displacement, significantly outperforming existing models like GNOT ($3.37 \times 10^{-1}$). - Computational Efficiency: Hilbert patching significantly lowered GPU memory requirements. In the
Shape-Net Cartask, increasing patch size from 1 to 32 reduced memory usage from ~60GB to under 10GB without sacrificing accuracy. - Interpretability: Attention heatmaps revealed that the decoder focuses on critical geometric boundaries and high-gradient regions, mimicking the focus areas of human engineering analysis.
Conclusion
MMET represents a significant step toward universal, real-time PDE solvers. By integrating spatial intelligence through Hilbert curves and robust condition handling via GCE, it provides a scalable solution for complex engineering problems. Future work will focus on developing MMET into a foundational pre-trained model for broader physics applications.
Summary updated