Introduction
Operator learning—the training of neural networks to map between infinite-dimensional function spaces—is a cornerstone of modern scientific machine learning. However, the computational cost and complexity of these models often lead to extensive hyperparameter tuning. In "Some Best Practices in Operator Learning," researchers Dustin Enyeart and Guang Lin from Purdue University conduct a systematic empirical study to establish robust training protocols for these specialized architectures.
Core Architectures
The study evaluates three primary neural operator architectures across various physical systems:
DeepONets
DeepONets utilize a two-pronged approach: a "branch" network that encodes the input function and a "trunk" network that encodes the domain coordinates. These are combined via a dot product to produce the final output.
Fourier Neural Operators (FNO)
FNOs leverage the Fourier transform to perform integral operations efficiently. By parameterizing kernels in the spectral domain, they can capture global dependencies in the data through spectral convolutions.
Koopman Autoencoders
These networks aim to identify a latent space where non-linear dynamics can be represented as linear transformations, governed by the Koopman operator.
Experimental Framework
The authors tested these architectures on five benchmark systems representing a broad range of physical phenomena:
- ODEs: The Pendulum (simple dynamics), the Lorenz System (chaotic dynamics), and a Fluid Attractor.
- PDEs: Burger’s Equation (fluid mechanics) and the Korteweg-de-Vries (KdV) Equation (solitons).
The sensitivity analysis focused on activation functions, dropout rates, Stochastic Weight Averaging (SWA), and automated learning rate finders.
Key Findings and Best Practices
1. Activation Functions: The Superiority of GeLU
The Gaussian Error Linear Unit (GeLU) was the undisputed champion, achieving the lowest error in 100% of the experiments. Because operator learning typically involves approximating smooth solutions to differential equations, the inherent smoothness of GeLU provides a better inductive bias than the "sharp" kink of a standard ReLU. In the Fluid Attractor test, GeLU's error was an order of magnitude lower than ReLU's.
2. The Negative Impact of Dropout
Contrary to its success in computer vision, dropout was found to be detrimental in operator learning. A dropout rate of 0.0 was optimal across all tests; increasing it to just 0.05 often doubled or tripled the error. The stochasticity introduced by dropout appears to interfere with the network's ability to resolve the precise continuous structures required for accurate physical modeling.
3. Stochastic Weight Averaging (SWA)
SWA can provide a significant accuracy boost if applied correctly. The key is to use a learning rate for the SWA phase that is equal to or lower than the base training rate. This allows the model to average weights within a stable local minimum rather than oscillating out of it.
4. Limitations of Automated LR Finders
Automated "Learning Rate Finders" (which look for the steepest descent in loss) performed poorly and inconsistently. The authors recommend manual tuning or standard schedules over these automated heuristics for operator learning tasks.
Conclusion
For researchers entering the field of operator learning, the "best practices" are clear: prioritize GeLU activations, eliminate dropout entirely, and use SWA with conservative learning rates. These simple adjustments can significantly reduce training time and improve the fidelity of neural solvers for complex physical systems.
Summary updated