Architecture-Aware Mixed Precision for Replica-Exchange Training of Geometry-Aware Neural Operators with Uncertainty Quantification

Ujin Jeon, Michael Penwarden, Wenqian Chen, Guang LinarXiv 2026 · arXiv

TL;DR

Architecture-Aware Mixed Precision (AA-MP) is a new training strategy that slashes the computational cost of geometry-aware neural operators by nearly 50% without sacrificing accuracy. By assigning high precision (FP32) to the "refinement" replica and low precision (BF16) to the "exploratory" replica in a replica-exchange framework, the authors achieve faster training and significantly lower GPU memory usage for complex physics simulations.

What Problem Does This Paper Address?

Scientific Machine Learning (SciML) relies heavily on neural operators—surrogate models that can learn to solve partial differential equations (PDEs) much faster than traditional solvers. However, when these models need to handle complex, varying shapes (like different jet engine brackets or airfoil designs), they require sophisticated architectures like the Arbitrary Geometry-encoded Transformer (ArGEnT).

These "geometry-aware" models are notoriously memory-hungry and computationally expensive due to their use of cross-attention mechanisms and large point-cloud representations. While standard Automatic Mixed Precision (AMP) exists, it often fails to balance the delicate trade-off between numerical stability and efficiency required for high-fidelity physics. The authors set out to find a way to make these models scalable for real-world engineering tasks.

Methods

The researchers combined two powerful concepts: Replica-Exchange Stochastic Gradient Langevin Dynamics (reSGLD) and a surgical application of Mixed Precision.

  1. Replica Exchange: The training process runs two versions (replicas) of the model simultaneously. A "cold" replica at a low "temperature" focuses on fine-tuning and local refinement, while a "hot" replica at a high temperature explores the broader optimization landscape to avoid getting stuck in local minima.
  2. AA-MP Strategy: Instead of applying mixed precision uniformly, the authors tailored it to the role of each replica:
  3. Cold Replica: Stays entirely in FP32 (32-bit float) to ensure the final model is as accurate as possible.
  4. Hot Replica: Uses BF16 (Brain Floating Point) for the heavy lifting—forward and backward passes through the Transformer layers.
  5. Architecture-Awareness: Crucially, they didn't just cast everything to low precision in the hot replica. They kept "master" parameters, optimizer states, and critical reduction operations (like the final fusion of physical parameters and geometry) in FP32 to maintain stability.
Figure 1: Overview of the proposed AA-MP ArGEnT–DeepONet training framework. (a) The DeepONet branch encodes non-geometric physical parameters, while the cross-attention ArGEnT trunk uses query features as queries and the geometry point cloud as keys and values. The exploratory hot replica uses a manually controlled BF16 live model, while the latent reduction, prediction-error and energy reductions, master parameters, and optimizer states are maintained in FP32. The cold replica executes the same network entirely in FP32. (b) Two replicas are evolved using preconditioned reSGLD at Tc < Th. The higher-temperature hot replica promotes broader exploration, whereas the FP32 cold replica performs low-temperature refinement and model evaluation. Replica-exchange proposals periodically exchange the parameter and associated optimizer states between the two replica slots.

Datasets

The authors evaluated their method on two challenging benchmarks:

  • AirfRANS: A fluid dynamics dataset involving 2D steady-state Reynolds-Averaged Navier-Stokes (RANS) solutions around various airfoil shapes.
  • Jet-Engine Bracket: A 3D structural mechanics dataset (derived from the DeepJEB benchmark) that predicts displacement and von Mises stress under different loading conditions.

Key Insights and Ideas

  • Role-Based Precision: The "aha" moment is that the exploratory (hot) replica doesn't need high precision to do its job. Its goal is to find promising regions of the loss landscape, not to provide the final answer. This asymmetry allows for massive resource savings.
  • Explicit Control Beats "Auto": By manually mapping which operations use which precision (as shown in Table 1), AA-MP outperforms standard PyTorch AMP. AMP often keeps too much in FP32 or performs unnecessary casts, whereas AA-MP is more aggressive where it counts.
  • Scaling Advantage: The memory benefits of AA-MP actually increase as you scale up the model (e.g., larger embedding dimensions) or the geometry (e.g., more points in the cloud). This makes it a vital tool for moving from toy problems to complex 3D engineering simulations.
Table 1: Precision structure of the proposed AA-MP configuration. The cold replica remains entirely in FP32. The table summarizes the manually controlled precision policy of the exploratory hot replica.

Key Results

  • Memory Savings: AA-MP reduced peak hot-replica GPU memory by 46.9% on both benchmarks compared to full FP32 training. It even beat standard AMP by about 9.4%.
  • Speedup: Training step time for the hot replica decreased by up to 25.8%.
  • Accuracy: Despite the lower precision in the hot replica, the final cold-replica model achieved accuracy comparable to (and sometimes slightly better than) the full-precision baseline.
  • Uncertainty Quantification (UQ): Using conformal prediction, the authors showed that AA-MP provides reliable uncertainty estimates with narrower, sharper prediction intervals than other mixed-precision methods, which is critical for safety-sensitive engineering applications.

Limitations and Future Work

  • Approximate Sampling: The exchange rule between replicas is approximate because it compares energies calculated at different precisions. While it works well for optimization, it isn't a strictly "exact" Bayesian sampling method.
  • Physics-Informed Models: The current work focuses on supervised learning. The authors suggest extending AA-MP to Physics-Informed Neural Networks (PINNs), where calculating higher-order derivatives for PDE residuals is even more computationally taxing.
  • Active Learning: Future research could integrate these calibrated uncertainty estimates with active learning to help the model decide which new geometries it needs to see to improve most efficiently.

Summary updated

Abstract

Geometry-aware neural operators face substantial computational and memory demands when learning across complex physical domains. We introduce Architecture-Aware Mixed Precision (AA-MP), a replica-exchange training strategy that allocates numerical precision according to replica roles and architecture-specific operations. By retaining FP32 for refinement while selectively employing BF16 for exploration, AA-MP preserves predictive accuracy while improving training efficiency. Across fluid dynamics and three-dimensional structural mechanics benchmarks, AA-MP achieves comparable accuracy to full-precision training over an ensemble of runs, while reducing hot-replica peak GPU memory by 46.9% and training time by up to 25.8%. It also outperforms naive Automatic Mixed Precision (AMP) in memory efficiency and computational cost, with increasing memory advantages at larger model and geometry scales. Ablation studies show that retaining FP32 in the cold-replica is important for predictive accuracy, while additional FP32 operations in the exploratory replica provide limited benefits relative to their computational cost. Finally, conformal prediction for uncertainty calibration shows that AA-MP achieves near-nominal coverage with narrower prediction intervals than the alternatives. These results advance the scalability of geometry-aware scientific machine learning by reducing computational costs without compromising predictive accuracy or uncertainty quantification.
BibTeX
@misc{jeon2026architecture,
  author = {Ujin Jeon and Michael Penwarden and Wenqian Chen and Guang Lin},
  title = {{Architecture-Aware Mixed Precision for Replica-Exchange Training of Geometry-Aware Neural Operators with Uncertainty Quantification}},
  year = {2026},
  eprint = {2611.02427},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02427},
}
Figure