DimOL: Dimensional Awareness as A New 'Dimension' in Operator Learning

Yichen Song, Yunbo Wang, Xiaokang Yang

Abstract

In the realm of computational physics, an enduring topic is the numerical solutions to partial differential equations (PDEs). Recently, the attention of researchers has shifted towards Neural Operator methods, renowned for their capability to approximate “operators” -- mappings from functions to functions. Despite the universal approximation theorem within neural operators, ensuring error bounds often requires employing numerous Fourier layers. However, what about lightweight models? In response to this question, we introduce DimOL (Dimension-aware Operator Learning), drawing insights from dimensional analysis. To implement DimOL, we propose the ProdLayer, which can be seamlessly integrated into FNO-based and Transformer-based PDE solvers, enhancing their ability to handle sum-of-products structures inherent in many physical systems. Empirically, DimOL models achieve up to 48% performance gain within the PDE datasets. Furthermore, by analyzing Fourier components' weights, we can symbolically discern the physical significance of each term. This sheds light on the opaque nature of neural networks, unveiling underlying physical principles.

Introduction

In the field of "AI for Science," Neural Operators, such as the Fourier Neural Operator (FNO), have emerged as powerful tools for solving partial differential equations (PDEs). However, these models often treat physical quantities—such as velocity, pressure, and viscosity—as dimensionless numbers. This oversight ignores fundamental physical constraints: one cannot add meters to seconds, and system behavior frequently depends on dimensionless ratios like the Reynolds number.

Current AI models' lack of dimensional awareness leads to poor generalization across different physical regimes. For instance, a model trained on low-viscosity fluids often fails when applied to high-viscosity scenarios. To mitigate this, researchers typically rely on massive, computationally expensive architectures. This work introduces DimINO (Dimension-Informed Neural Operators), a framework that embeds classical dimensional analysis into neural operator learning to create lightweight, physically consistent models.

Methodology: Dimension-Informed Architecture

DimINO introduces two "plug-and-play" modules that can be integrated into existing neural operator architectures without increasing the parameter count. These modules ensure that the model respects physical units and satisfies Similar Transformation Invariance (STI)—the principle that underlying physics should remain invariant under unit rescaling.

The DimINO architecture featuring DimNorm, Neural Operator Blocks, and ReDimensionalization modules.
  1. DimNorm: Instead of standard normalization, this module utilizes dimensionless numbers derived from the PDE's physical constants to normalize input fields. A "DimGate" mechanism—a soft gating system—is employed to inject these physical priors into the latent space.
  2. ReDimensionalization: After the neural operator processes the dimensionless data, this module maps the results back into physical fields with the correct units, ensuring output consistency with the input.

The authors prove a Universal Approximation Theorem for DimINO, demonstrating that it can effectively approximate operators while maintaining physical sanity.

Experimental Results

The researchers evaluated DimINO against the Tensorized Fourier Neural Operator (T-FNO) baseline using several classic PDE benchmarks, including 1D Advection, 1D Burgers, and 2D Diffusion-Reaction equations.

Performance Gains

Empirical results show significant error reductions across the board. On the complex 2D Diffusion-Reaction task, DimINO reduced the relative L2 error by 76.3% compared to the T-FNO baseline. It also achieved a 31.6% gain on Advection and 12.5% on Burgers equations.

The STI Stress Test

A critical advantage of DimINO is its robustness to unit scaling. In a "stress test" where inputs were rescaled by a factor $p$, the baseline T-FNO's error exploded as $p$ increased, indicating a failure to generalize to out-of-distribution scales. In contrast, DimINO’s error remained perfectly flat, proving it had successfully learned scale-invariant physics.

Comparison of relative L2 error under different scaling factors p, demonstrating DimINO's scale invariance compared to the T-FNO baseline.

Conclusion

DimINO demonstrates that dimensional information should not merely be appended to inputs but used to modulate features in a principled way. By ensuring models see the "dimensionless" version of a problem, they become immune to scaling issues that plague standard architectures. While the benefits are most pronounced when physical constants are known, DimINO provides a robust framework for building physically-aware AI models that are both lightweight and highly accurate. Future work will explore integrating these principles into broader architectures like Transformer-based operators and Graph Neural Networks.

Summary updated

Abstract

In the realm of computational physics, an enduring topic is the numerical solutions to partial differential equations (PDEs). Recently, the attention of researchers has shifted towards Neural Operator methods, renowned for their capability to approximate ``operators'' -- mappings from functions to functions. Despite the universal approximation theorem within neural operators, ensuring error bounds often requires employing numerous Fourier layers. However, what about lightweight models? In response to this question, we introduce DimOL (Dimension-aware Operator Learning), drawing insights from dimensional analysis. To implement DimOL, we propose the ProdLayer, which can be seamlessly integrated into FNO-based and Transformer-based PDE solvers, enhancing their ability to handle sum-of-products structures inherent in many physical systems. Empirically, DimOL models achieve up to 48% performance gain within the PDE datasets. Furthermore, by analyzing Fourier components' weights, we can symbolically discern the physical significance of each term. This sheds light on the opaque nature of neural networks, unveiling underlying physical principles.
BibTeX
@misc{song2024dimol,
  author = {Yichen Song and Yunbo Wang and Xiaokang Yang},
  title = {{DimOL: Dimensional Awareness as A New 'Dimension' in Operator Learning}},
  year = {2024},
  eprint = {2410.05894},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.05894},
}
Figure