Some Best Practices in Operator Learning

Guang Lin, Dustin Enyeart, Guang Lin 0001arXiv 2024 · DOI · arXiv

Introduction

Operator learning—the training of neural networks to map between infinite-dimensional function spaces—is a cornerstone of modern scientific machine learning. However, the computational cost and complexity of these models often lead to extensive hyperparameter tuning. In "Some Best Practices in Operator Learning," researchers Dustin Enyeart and Guang Lin from Purdue University conduct a systematic empirical study to establish robust training protocols for these specialized architectures.

Core Architectures

The study evaluates three primary neural operator architectures across various physical systems:

DeepONets

DeepONets utilize a two-pronged approach: a "branch" network that encodes the input function and a "trunk" network that encodes the domain coordinates. These are combined via a dot product to produce the final output.

Schematic of the DeepONet architecture, featuring branch and trunk networks.

Fourier Neural Operators (FNO)

FNOs leverage the Fourier transform to perform integral operations efficiently. By parameterizing kernels in the spectral domain, they can capture global dependencies in the data through spectral convolutions.

Internal structure of a Fourier Neural Operator (FNO) layer.

Koopman Autoencoders

These networks aim to identify a latent space where non-linear dynamics can be represented as linear transformations, governed by the Koopman operator.

Experimental Framework

The authors tested these architectures on five benchmark systems representing a broad range of physical phenomena:

  • ODEs: The Pendulum (simple dynamics), the Lorenz System (chaotic dynamics), and a Fluid Attractor.
  • PDEs: Burger’s Equation (fluid mechanics) and the Korteweg-de-Vries (KdV) Equation (solitons).

The sensitivity analysis focused on activation functions, dropout rates, Stochastic Weight Averaging (SWA), and automated learning rate finders.

Key Findings and Best Practices

1. Activation Functions: The Superiority of GeLU

The Gaussian Error Linear Unit (GeLU) was the undisputed champion, achieving the lowest error in 100% of the experiments. Because operator learning typically involves approximating smooth solutions to differential equations, the inherent smoothness of GeLU provides a better inductive bias than the "sharp" kink of a standard ReLU. In the Fluid Attractor test, GeLU's error was an order of magnitude lower than ReLU's.

2. The Negative Impact of Dropout

Contrary to its success in computer vision, dropout was found to be detrimental in operator learning. A dropout rate of 0.0 was optimal across all tests; increasing it to just 0.05 often doubled or tripled the error. The stochasticity introduced by dropout appears to interfere with the network's ability to resolve the precise continuous structures required for accurate physical modeling.

3. Stochastic Weight Averaging (SWA)

SWA can provide a significant accuracy boost if applied correctly. The key is to use a learning rate for the SWA phase that is equal to or lower than the base training rate. This allows the model to average weights within a stable local minimum rather than oscillating out of it.

4. Limitations of Automated LR Finders

Automated "Learning Rate Finders" (which look for the steepest descent in loss) performed poorly and inconsistently. The authors recommend manual tuning or standard schedules over these automated heuristics for operator learning tasks.

Conclusion

For researchers entering the field of operator learning, the "best practices" are clear: prioritize GeLU activations, eliminate dropout entirely, and use SWA with conservative learning rates. These simple adjustments can significantly reduce training time and improve the fidelity of neural solvers for complex physical systems.

Summary updated

BibTeX
@misc{lin2024some,
  author = {Guang Lin and Dustin Enyeart and Guang Lin},
  title = {{Some Best Practices in Operator Learning}},
  year = {2024},
  eprint = {2412.06686},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.06686},
}
Figure