MMET: A Multi-Input and Multi-Scale Transformer for Efficient PDEs Solving

Yichen Luo, Jia Wang, Dapeng Lan, Yu Liu, Zhibo Pang

Abstract

Solving Partial Differential Equations (PDEs) is fundamental to engineering but remains computationally intensive for traditional numerical methods. We present the Multi-input and Multi-scale Efficient Transformer (MMET), a novel framework designed to solve PDEs across diverse geometries, boundary conditions, and resolutions. By decoupling mesh encoding from point queries and employing Hilbert curve-based patching, MMET addresses the challenges of multi-input generalization and computational scaling. Our results demonstrate state-of-the-art accuracy and efficiency on complex 2D and 3D engineering benchmarks.

Introduction: The Challenge of PDE Solving

Partial Differential Equations (PDEs) underpin nearly all physical simulations, from aerodynamics to thermodynamics. While traditional methods like the Finite Element Method (FEM) provide high accuracy, their computational cost often prohibits real-time application. Neural Operators have emerged as a faster alternative, yet they frequently struggle with three core issues: generalizing across different physical boundary conditions, handling multi-scale resolutions, and scaling to massive 3D meshes without memory exhaustion.

Overview of the MMET framework addressing engineering challenges.

The MMET Framework

To overcome these hurdles, MMET introduces three primary architectural innovations:

1. Encoder-Decoder Decoupling

Standard Transformers often process inputs as a single sequence. MMET instead treats the geometric and physical mesh as the "source" for the Encoder and the specific coordinates for which a solution is required as "queries" for the Decoder. This decoupling enables the model to perform zero-shot resolution scaling, predicting values at any spatial point regardless of the training resolution.

2. Gated Condition Embedding (GCE)

Handling diverse physical inputs requires distinguishing between actual zero-valued measurements and missing data. The GCE layer utilizes a gating mechanism to resolve this "ambiguity of zero," allowing a single model to process Dirichlet, Neumann, and Robin boundary conditions simultaneously.

3. Hilbert Curve Reserialization & Patching

For large-scale models, MMET maps 2D/3D points into a 1D sequence using a Hilbert curve. This approach preserves spatial locality better than standard scanning methods. These sequences are then "patched" into tokens, drastically reducing the sequence length for the Transformer's attention mechanism while maintaining local geometric fidelity.

Detailed architecture of MMET showing the flow from mesh input to simulation results.

Experimental Evaluation

MMET was evaluated on six benchmarks, including Poisson, Darcy Flow, Shape-Net Car (3D aerodynamics), and various heat and stress problems (Heat2d, Beam2d, HeatSink2d).

Key Results

  • Superior Accuracy: On the complex Beam2d dataset, MMET achieved a relative $L_2$ error of $4.64 \times 10^{-2}$ for displacement, significantly outperforming existing models like GNOT ($3.37 \times 10^{-1}$).
  • Computational Efficiency: Hilbert patching significantly lowered GPU memory requirements. In the Shape-Net Car task, increasing patch size from 1 to 32 reduced memory usage from ~60GB to under 10GB without sacrificing accuracy.
  • Interpretability: Attention heatmaps revealed that the decoder focuses on critical geometric boundaries and high-gradient regions, mimicking the focus areas of human engineering analysis.

Conclusion

MMET represents a significant step toward universal, real-time PDE solvers. By integrating spatial intelligence through Hilbert curves and robust condition handling via GCE, it provides a scalable solution for complex engineering problems. Future work will focus on developing MMET into a foundational pre-trained model for broader physics applications.

Summary updated

Abstract

Partial Differential Equations (PDEs) are fundamental for modeling physical systems, yet solving them in a generic and efficient manner using machine learning-based approaches remains challenging due to limited multi-input and multi-scale generalization capabilities, as well as high computational costs. This paper proposes the Multi-input and Multi-scale Efficient Transformer (MMET), a novel framework designed to address the above challenges. MMET decouples mesh and query points as two sequences and feeds them into the encoder and decoder, respectively, and uses a Gated Condition Embedding (GCE) layer to embed input variables or functions with varying dimensions, enabling effective solutions for multi-scale and multi-input problems. Additionally, a Hilbert curve-based reserialization and patch embedding mechanism decrease the input length. This significantly reduces the computational cost when dealing with large-scale geometric models. These innovations enable efficient representations and support multi-scale resolution queries for large-scale and multi-input PDE problems. Experimental evaluations on diverse benchmarks spanning different physical fields demonstrate that MMET outperforms SOTA methods in both accuracy and computational efficiency. This work highlights the potential of MMET as a robust and scalable solution for real-time PDE solving in engineering and physics-based applications, paving the way for future explorations into pre-trained large-scale models in specific domains. This work is open-sourced at https://github.com/YichenLuo-0/MMET.
BibTeX
@inproceedings{luo2025mmet,
  author = {Yichen Luo and Jia Wang and Dapeng Lan and Yu Liu and Zhibo Pang},
  title = {{MMET: A Multi-Input and Multi-Scale Transformer for Efficient PDEs Solving}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=fHgJueZOiA},
}
Figure