DimOL: Dimensional Awareness as a New 'Dimension' in Operator Learning

Yichen Song, Yunbo Wang, Xiaokang YangSubmission to ICLR 2025 · Withdrawn

TL;DR

Neural operators for partial differential equations (PDEs) often struggle when multiple physical background conditions vary, partly because standard network architectures ignore the dimensional rules governing physics. The authors introduce DimOL (Dimension-Aware Operator Learning) and a lightweight ProdLayer that explicitly encodes product terms between latent channels. Integrating ProdLayer into Fourier Neural Operators (FNO) and Transformer-based models cuts prediction errors by up to 48%, improves out-of-distribution generalization under physical scaling laws, and aids in reverse-engineering symbolic physical derivatives.

What Problem Does This Paper Address?

While Neural Operators (such as Fourier Neural Operators and Transformer-based PDE solvers) have become popular for mapping between continuous function spaces, they often demand deep, parameter-heavy architectures to achieve acceptable error bounds. Furthermore, when physical environments vary—such as changing viscosity or fluctuating external forces—standard models struggle to generalize.

A fundamental cause is that typical deep learning building blocks (linear projections, standard MLPs) are dimensionally agnostic: they naively add features together. However, basic physics dictates that quantities with different units/dimensions cannot be added, but they can be sensibly multiplied. Because governing dynamical equations (e.g., Navier–Stokes) and analytical solutions (e.g., separation of variables) inherently feature sums of products, models lacking explicit multiplicative mechanisms struggle to learn these relationships without overfitting.

Methods

The paper introduces DimOL (Dimension-Aware Operator Learning), realized through a drop-in architectural unit called ProdLayer:

  • ProdLayer Formulation: Given an input feature vector split into sub-vectors $x = x_a \oplus x_b \oplus x_2$ (where $x_a, x_b \in \mathbb{R}^p$ and $p$ is the number of product pairs), the layer calculates: $$\sigma_{\text{prod}}(x) = W [(x_a \otimes x_b) \oplus x_2]$$ Here, $\otimes$ denotes an element-wise product and $W$ is a learnable linear transform.

  • Model Integration:

  • In FNO/T-FNO, ProdLayer replaces the standard channel-mixing MLP following the spectral convolution.
  • In Transformer-based models (LSM, GNOT), ProdLayer replaces linear projection layers within the decoders, enabling the network to directly multiply coordinate basis functions (like sine/cosine) rather than merely shifting them with additive biases.
  • Dimensional Placement: The authors emphasize that dimensional awareness is not about adding arbitrary products anywhere. For example, replacing Fourier-domain linear transforms with ProdLayers results in spatial convolutions ($f * g$) without clear physical rationale. ProdLayer is specifically placed where channel interactions mimic polynomial and sum-of-products physical operations.

Datasets

The authors evaluated DimOL across both regular and irregular mesh datasets:

  • 1D Burgers' Equation: Classic non-linear benchmark.
  • 2D Darcy Flow: Steady-state fluid flow through porous media.
  • 2D Navier–Stokes:
  • TorusLi: Standard periodic vorticity benchmark with fixed external force.
  • TorusVisForce: Introduced by Tran et al. (2023), varying both viscosity and forcing terms across simulations. Subsets (TorusVisForce-TopMu and -LowMu) were constructed for few-shot and out-of-distribution (OOD) evaluations.
  • Inductor2D: An irregular-mesh electromagnetics dataset from the GNOT benchmark involving varying physical quantities (as opposed to geometry-only meshes like Airfoil, which lack input physical dimensions).

Key Insights and Ideas

  • Multiplication over Addition for Physical Quantities: Standard MLPs often overfit when acting as channel mixers on physical fields. In contrast, ProdLayer acts as a natural regularizer by providing an inductive bias for product terms (e.g., convective terms $v \cdot \nabla v$).
  • Better than Indirect Polynomial Formulations: Attempting to represent products through square differences ($ab = \frac{1}{4}[(a+b)^2 - (a-b)^2]$) with quadratic layers underperformed compared to direct element-wise products.
  • Invariance to Similar Transformations (Partial-OOD): Dimensional analysis dictates that scaling physical parameters systematically ($\omega_0 \to k\omega_0$, $t \to t/k$, $\nu \to k\nu$, $f \to k^2 f$) preserves governing dynamics. DimOL models generalize dramatically better than base models when subjected to these transformed, out-of-distribution inputs.
  • Symbolic Interpretability: By inspecting the Fourier weights in a 1-layer FNO + ProdLayer model trained on Burgers' equation, the network's weights directly regressed linear ($ik$) and quadratic ($-k^2$) spectral operators, perfectly recovering the first ($\partial_x$) and second ($\partial_{xx}$) spatial derivatives.

Key Results

  • 1D Burgers: Adding ProdLayer to T-FNO reduced test MSE by 48.4% (from $2.225 \times 10^{-3}$ to $1.147 \times 10^{-3}$), while adding a standard MLP channel mixer actually degraded performance by 5.3% due to overfitting.
  • Darcy Flow: Consistent MSE drops across all models: 2.8% for FNO, 7.8% for T-FNO, and 6.2% for LSM.
  • Navier–Stokes (TorusLi & TorusVisForce):
  • On TorusLi, FNO + ProdLayer outperformed the baseline by 6.8%.
  • On TorusVisForce, T-FNO + ProdLayer lowered MSE by 11.5% for short-horizon ($T=4$) and 5.5% for long-horizon ($T=10$) forecasting.
  • In few-shot testing ($N=100$), T-FNO + ProdLayer attained a 10.4% in-distribution gain and up to 11.6% on scaled partial-OOD evaluations.
  • Inductor2D (Irregular Mesh): GNOT + ProdLayer improved performance by 4.8% over baseline GNOT with zero hyperparameter tuning ($p=1$).

Limitations and Future Work

  • Dependence on Physical Inputs: DimOL requires inputs with meaningful physical dimensions. It provides no benefit on geometry-only or purely coordinate-based datasets where physical parameters are not fed to the model.
  • Simple Product Assumption: ProdLayer explicitly models low-order products ($x_a \otimes x_b$). For systems governed by highly complex, non-polynomial, or transcendental relationships between dimensional groups, simple pairwise multiplications may not fully capture the underlying physics.
  • Theoretical Formalization: While empirically verified and physically intuitive, formally quantifying "dimensional awareness" in operator learning remains an open problem.

Summary updated

Abstract

In the realm of computational physics, an enduring topic is the numerical solutions to partial differential equations (PDEs). Recently, the attention of researchers has shifted towards Neural Operator methods, renowned for their capability to approximate "operators'' --- mappings from functions to functions. Despite the universal approximation theorem within neural operators, ensuring error bounds often requires employing numerous Fourier layers. However, what about lightweight models? In response to this question, we introduce DimOL (Dimension-aware Operator Learning), drawing insights from dimensional analysis. To implement DimOL, we propose the ProdLayer, which can be seamlessly integrated into FNO-based and Transformer-based PDE solvers, enhancing their ability to handle sum-of-products structures inherent in many physical systems. Empirically, DimOL models achieve up to 48% performance gain within the PDE datasets. Furthermore, by analyzing Fourier components' weights, we can symbolically discern the physical significance of each term. This sheds light on the opaque nature of neural networks, unveiling underlying physical principles.
BibTeX
@misc{song2024dimol,
  author = {Yichen Song and Yunbo Wang and Xiaokang Yang},
  title = {{DimOL: Dimensional Awareness as a New 'Dimension' in Operator Learning}},
  year = {2024},
  eprint = {2410.05894},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.05894},
}

Loading related content...

Figure