Generative flow induced neural architecture search: Towards discovering optimal architecture in wavelet neural operator

Hartej Soin, Tapas Tripura, Souvik Chakraborty

Abstract

We propose a generative flow-induced neural architecture search algorithm. The proposed approach devices simple feed-forward neural networks to learn stochastic policies to generate sequences of architecture hyperparameters such that the generated states are in proportion with the reward from the terminal state. We demonstrate the efficacy of the proposed search algorithm on the wavelet neural operator (WNO), where we learn a policy to generate a sequence of hyperparameters like wavelet basis and activation operators for wavelet integral blocks. While the trajectory of the generated wavelet basis and activation sequence is cast as flow, the policy is learned by minimizing the flow violation between each state in the trajectory and maximizing the reward from the terminal state. In the terminal state, we train WNO simultaneously to guide the search. We propose to use the exponent of the negative of the WNO loss on the validation dataset as the reward function. While the grid search-based neural architecture generation algorithms foresee every combination, the proposed framework generates the most probable sequence based on the positive reward from the terminal state, thereby reducing exploration time. Compared to reinforcement learning schemes, where complete episodic training is required to get the reward, the proposed algorithm generates the hyperparameter trajectory sequentially. Through four fluid mechanics-oriented problems, we illustrate that the learned policies can sample the best-performing architecture of the neural operator, thereby improving the performance of the vanilla wavelet neural operator.

Introduction

Solving Partial Differential Equations (PDEs) is a cornerstone of scientific computing, yet traditional numerical methods often struggle with high computational costs and grid dependency. Neural Operators, such as the Wavelet Neural Operator (WNO), have emerged as a promising solution, offering discretization-invariant learning of physical operators. However, the performance of WNOs is highly sensitive to the choice of wavelet bases and activation functions across its layers. Manually searching this vast hyperparameter space is inefficient and often leads to sub-optimal global choices. This paper introduces FWNO (Flow-induced Wavelet Neural Operator), a framework that leverages Generative Flow Networks (GFlowNets) to automate the discovery of optimal WNO architectures.

Methodology: Generative Flow-induced NAS

The core of FWNO is a search algorithm that treats neural architecture construction as a probabilistic flow problem. Instead of a standard grid search or reinforcement learning approach, the authors utilize GFlowNets to learn a stochastic policy for generating sequences of hyperparameters.

Architecture as a Trajectory

The process of building a WNO is modeled as a sequence of actions. For each Wavelet Integral Block (WIB), the agent selects a wavelet basis ($w$) and an activation function ($a$). This sequence forms a trajectory through the state space, starting from a root node and progressing through $l$ layers.

Overview of the FWNO framework, showing (a) the flow through states for architecture search, (b) the overall WNO architecture, and (c) the internal structure of the Wavelet Integral Block (WIB).

Flow Consistency and Reward

Unlike Reinforcement Learning, which often focuses on finding a single optimal path, GFlowNets learn to sample trajectories with probabilities proportional to a terminal reward. The reward function is defined as $R = e^{-Loss_{val}}$, where $Loss_{val}$ is the validation loss of the generated WNO architecture. Two feed-forward networks, $N_w$ and $N_a$, are trained to minimize flow violations between states while maximizing this reward. This allows for efficient exploration and convergence on high-performing architectures without the need for exhaustive search.

Experimental Results

The FWNO framework was evaluated on four benchmark fluid mechanics problems: 1D Burgers’ equation, 2D Darcy flow (on both square and notched domains), and the 2D incompressible Navier-Stokes equation.

Performance Gains

In all cases, the automatically discovered architectures outperformed "vanilla" WNO models that used fixed, manually selected hyperparameters. The improvements were particularly significant in complex 2D simulations.

PDE Example Vanilla WNO Error FWNO Error Improvement
1D Burgers’ 1.75% 1.44% ~18% reduction
2D Darcy (Square) 1.80% 1.58% ~12% reduction
2D Darcy (Triangular) 0.88% 0.59% ~33% reduction
2D Navier-Stokes 3.43% 2.35% ~31% reduction

Key Insights

  • Layer-Specific Optimization: The search revealed that optimal performance often requires different wavelet/activation pairs for different layers. For instance, in Burgers' equation, the model found that the first layer performed best with a bior6.8 wavelet, while subsequent layers preferred db6.
  • Zero-Shot Super-Resolution: The discovered architectures maintained the WNO's discretization invariance, performing well on high-resolution grids even when the architecture search was conducted on lower resolutions.
  • Efficiency: By generating sequences based on positive rewards, FWNO significantly reduces the time required for architecture exploration compared to exhaustive grid searches.

Conclusion

FWNO demonstrates that Generative Flow Networks provide a powerful and efficient mechanism for Neural Architecture Search in the context of operator learning. By automating the selection of wavelet bases and activations, the framework significantly enhances the accuracy of WNOs across diverse physical simulations, providing a robust path forward for automated scientific machine learning.

Summary updated

Abstract

We propose a generative flow-induced neural architecture search algorithm. The proposed approach devices simple feed-forward neural networks to learn stochastic policies to generate sequences of architecture hyperparameters such that the generated states are in proportion with the reward from the terminal state. We demonstrate the efficacy of the proposed search algorithm on the wavelet neural operator (WNO), where we learn a policy to generate a sequence of hyperparameters like wavelet basis and activation operators for wavelet integral blocks. While the trajectory of the generated wavelet basis and activation sequence is cast as flow, the policy is learned by minimizing the flow violation between each state in the trajectory and maximizing the reward from the terminal state. In the terminal state, we train WNO simultaneously to guide the search. We propose to use the exponent of the negative of the WNO loss on the validation dataset as the reward function. While the grid search-based neural architecture generation algorithms foresee every combination, the proposed framework generates the most probable sequence based on the positive reward from the terminal state, thereby reducing exploration time. Compared to reinforcement learning schemes, where complete episodic training is required to get the reward, the proposed algorithm generates the hyperparameter trajectory sequentially. Through four fluid mechanics-oriented problems, we illustrate that the learned policies can sample the best-performing architecture of the neural operator, thereby improving the performance of the vanilla wavelet neural operator.
BibTeX
@article{soin2025generative,
  author = {Hartej Soin and Tapas Tripura and Souvik Chakraborty},
  title = {{Generative flow induced neural architecture search: Towards discovering optimal architecture in wavelet neural operator}},
  journal = {Comput. Phys. Commun},
  year = {2025},
  doi = {10.1016/j.cpc.2025.109755},
}
Figure