Generative flow induced neural architecture search: Towards discovering optimal architecture in wavelet neural operator

Hartej Soin, Tapas Tripura, Souvik Chakraborty

TL;DR

This paper introduces FWNO (Flow-induced Wavelet Neural Operator), a framework that uses Generative Flow Networks (GFlowNets) to automatically discover the optimal architecture for Wavelet Neural Operators. By treating neural architecture search as a probabilistic flow problem, the authors successfully automate the selection of wavelet bases and activation functions, leading to significant accuracy gains across complex fluid mechanics simulations.

What Problem Does This Paper Address?

Solving Partial Differential Equations (PDEs) is fundamental to science and engineering, but traditional methods like Finite Element Analysis are computationally expensive and tied to specific grids. Neural Operators (like FNO and WNO) have emerged as powerful, "discretization-invariant" alternatives that can learn the underlying physics once and predict solutions on any mesh resolution.

However, Wavelet Neural Operators (WNOs) come with a headache: they require choosing specific wavelet bases (e.g., Daubechies, Symlets) and activation functions for every single layer. Usually, researchers just pick one combination and apply it globally, or perform an exhaustive "grid search" that wastes massive amounts of time and compute. This paper tackles the challenge of efficiently searching this vast hyperparameter space to find the best-performing architecture for a given physical problem.

Methods

The authors propose a Generative Flow-induced Neural Architecture Search algorithm. The core innovation is the use of GFlowNets:

  1. Architecture as a Trajectory: The process of building a WNO is treated as a sequence of actions. For each layer, the agent chooses a wavelet basis and an activation function.
  2. Flow Consistency: Unlike Reinforcement Learning (RL), which typically tries to find a single "optimal" path, GFlowNets learn a policy to sample trajectories with a probability proportional to a reward. The "flow" into a state (a partial architecture) must match the "flow" out of it.
  3. The Reward Function: The reward is defined as the exponent of the negative validation loss ($R = e^{-Loss}$). This ensures that architectures that result in lower error are sampled more frequently.
  4. Training: Two simple feed-forward networks ($N_w$ and $N_a$) learn the policy. They are trained by minimizing the discrepancy between inflow and outflow while maximizing the terminal reward. This allows the system to explore diverse architectures and converge on the most effective ones without checking every possible combination.

Datasets

The framework was evaluated on four classic benchmarks in fluid mechanics and structural analysis:

  • 1D Burgers’ Equation: A fundamental PDE for modeling fluid flow, traffic, and acoustics.
  • 2D Darcy Flow (Square Domain): Modeling fluid flow through porous media (like water through soil).
  • 2D Darcy Flow (Triangular Domain with a Notch): A more complex geometry designed to test the model's robustness to irregular boundaries.
  • 2D Incompressible Navier-Stokes Equation: The "gold standard" for simulating viscous fluid dynamics (vorticity over time).

Key Insights and Ideas

  • Layer-Specific Optimization: A major "aha" moment is that the optimal wavelet/activation pair isn't the same for every layer. For example, in the Burgers' equation test, the model found that the first layer performed best with a bior6.8 wavelet, while subsequent layers preferred db6.
  • Efficiency over Grid Search: Grid search is "blind"—it checks every combination regardless of performance. FWNO generates the most probable sequences based on positive rewards, significantly reducing exploration time.
  • Probabilistic Diversity: Because it's based on GFlowNets, the search doesn't just collapse to one point; it learns the distribution of "good" architectures. This is useful for understanding which hyperparameters are truly critical for a specific type of physics.
  • Preserving Invariance: The search process doesn't break the WNO's best feature: discretization invariance. An architecture found on a low-resolution grid still performs exceptionally well when tested on much higher resolutions (zero-shot super-resolution).

Key Results

The FWNO consistently outperformed the "vanilla" WNO (which uses a fixed, manually chosen architecture) across all tests.

PDE Example Vanilla WNO Error FWNO Error Improvement
1D Burgers’ 1.75% 1.44% ~18% reduction
2D Darcy (Square) 1.80% 1.58% ~12% reduction
2D Darcy (Triangular) 0.88% 0.59% ~33% reduction
2D Navier-Stokes 3.43% 2.35% ~31% reduction

Notably, in the Navier-Stokes case, the error dropped by over a full percentage point, which is a significant margin in high-precision scientific modeling.

Limitations and Future Work

  • Search Space Scope: Currently, the search is limited to wavelet bases and activation functions. The authors note that future work could incorporate other hyperparameters, such as the number of convolution layers or the depth of wavelet decomposition.
  • Computational Overhead: While more efficient than grid search, training the GFlowNet agents still adds an initial overhead compared to just training a single fixed model.
  • Generalization: The current study focuses on WNOs. While the authors suggest the strategy can be integrated with other neural operators (like FNO or DeepONet), this hasn't been demonstrated in this specific paper.

Summary updated

Abstract

We propose a generative flow-induced neural architecture search algorithm. The proposed approach devices simple feed-forward neural networks to learn stochastic policies to generate sequences of architecture hyperparameters such that the generated states are in proportion with the reward from the terminal state. We demonstrate the efficacy of the proposed search algorithm on the wavelet neural operator (WNO), where we learn a policy to generate a sequence of hyperparameters like wavelet basis and activation operators for wavelet integral blocks. While the trajectory of the generated wavelet basis and activation sequence is cast as flow, the policy is learned by minimizing the flow violation between each state in the trajectory and maximizing the reward from the terminal state. In the terminal state, we train WNO simultaneously to guide the search. We propose to use the exponent of the negative of the WNO loss on the validation dataset as the reward function. While the grid search-based neural architecture generation algorithms foresee every combination, the proposed framework generates the most probable sequence based on the positive reward from the terminal state, thereby reducing exploration time. Compared to reinforcement learning schemes, where complete episodic training is required to get the reward, the proposed algorithm generates the hyperparameter trajectory sequentially. Through four fluid mechanics-oriented problems, we illustrate that the learned policies can sample the best-performing architecture of the neural operator, thereby improving the performance of the vanilla wavelet neural operator.
BibTeX
@article{soin2025generative,
  author = {Hartej Soin and Tapas Tripura and Souvik Chakraborty},
  title = {{Generative flow induced neural architecture search: Towards discovering optimal architecture in wavelet neural operator}},
  journal = {Comput. Phys. Commun},
  year = {2025},
  doi = {10.1016/j.cpc.2025.109755},
}

Loading related content...

Figure