Abstract
We propose a generative flow-induced neural architecture search algorithm. The proposed approach devices simple feed-forward neural networks to learn stochastic policies to generate sequences of architecture hyperparameters such that the generated states are in proportion with the reward from the terminal state. We demonstrate the efficacy of the proposed search algorithm on the wavelet neural operator (WNO), where we learn a policy to generate a sequence of hyperparameters like wavelet basis and activation operators for wavelet integral blocks. While the trajectory of the generated wavelet basis and activation sequence is cast as flow, the policy is learned by minimizing the flow violation between each state in the trajectory and maximizing the reward from the terminal state. In the terminal state, we train WNO simultaneously to guide the search. We propose to use the exponent of the negative of the WNO loss on the validation dataset as the reward function. While the grid search-based neural architecture generation algorithms foresee every combination, the proposed framework generates the most probable sequence based on the positive reward from the terminal state, thereby reducing exploration time. Compared to reinforcement learning schemes, where complete episodic training is required to get the reward, the proposed algorithm generates the hyperparameter trajectory sequentially. Through four fluid mechanics-oriented problems, we illustrate that the learned policies can sample the best-performing architecture of the neural operator, thereby improving the performance of the vanilla wavelet neural operator.
Introduction
Solving Partial Differential Equations (PDEs) is a cornerstone of scientific computing, yet traditional numerical methods often struggle with high computational costs and grid dependency. Neural Operators, such as the Wavelet Neural Operator (WNO), have emerged as a promising solution, offering discretization-invariant learning of physical operators. However, the performance of WNOs is highly sensitive to the choice of wavelet bases and activation functions across its layers. Manually searching this vast hyperparameter space is inefficient and often leads to sub-optimal global choices. This paper introduces FWNO (Flow-induced Wavelet Neural Operator), a framework that leverages Generative Flow Networks (GFlowNets) to automate the discovery of optimal WNO architectures.
Methodology: Generative Flow-induced NAS
The core of FWNO is a search algorithm that treats neural architecture construction as a probabilistic flow problem. Instead of a standard grid search or reinforcement learning approach, the authors utilize GFlowNets to learn a stochastic policy for generating sequences of hyperparameters.
Architecture as a Trajectory
The process of building a WNO is modeled as a sequence of actions. For each Wavelet Integral Block (WIB), the agent selects a wavelet basis ($w$) and an activation function ($a$). This sequence forms a trajectory through the state space, starting from a root node and progressing through $l$ layers.
Flow Consistency and Reward
Unlike Reinforcement Learning, which often focuses on finding a single optimal path, GFlowNets learn to sample trajectories with probabilities proportional to a terminal reward. The reward function is defined as $R = e^{-Loss_{val}}$, where $Loss_{val}$ is the validation loss of the generated WNO architecture. Two feed-forward networks, $N_w$ and $N_a$, are trained to minimize flow violations between states while maximizing this reward. This allows for efficient exploration and convergence on high-performing architectures without the need for exhaustive search.
Experimental Results
The FWNO framework was evaluated on four benchmark fluid mechanics problems: 1D Burgers’ equation, 2D Darcy flow (on both square and notched domains), and the 2D incompressible Navier-Stokes equation.
Performance Gains
In all cases, the automatically discovered architectures outperformed "vanilla" WNO models that used fixed, manually selected hyperparameters. The improvements were particularly significant in complex 2D simulations.
| PDE Example | Vanilla WNO Error | FWNO Error | Improvement |
|---|---|---|---|
| 1D Burgers’ | 1.75% | 1.44% | ~18% reduction |
| 2D Darcy (Square) | 1.80% | 1.58% | ~12% reduction |
| 2D Darcy (Triangular) | 0.88% | 0.59% | ~33% reduction |
| 2D Navier-Stokes | 3.43% | 2.35% | ~31% reduction |
Key Insights
- Layer-Specific Optimization: The search revealed that optimal performance often requires different wavelet/activation pairs for different layers. For instance, in Burgers' equation, the model found that the first layer performed best with a
bior6.8wavelet, while subsequent layers preferreddb6. - Zero-Shot Super-Resolution: The discovered architectures maintained the WNO's discretization invariance, performing well on high-resolution grids even when the architecture search was conducted on lower resolutions.
- Efficiency: By generating sequences based on positive rewards, FWNO significantly reduces the time required for architecture exploration compared to exhaustive grid searches.
Conclusion
FWNO demonstrates that Generative Flow Networks provide a powerful and efficient mechanism for Neural Architecture Search in the context of operator learning. By automating the selection of wavelet bases and activations, the framework significantly enhances the accuracy of WNOs across diverse physical simulations, providing a robust path forward for automated scientific machine learning.
Summary updated