Choosing Fourier Modes by What Predicts the Output, Not by Frequency

A Fourier neural operator learns a map from one field to another, for example from a source distribution to the wavefield it produces. It works in the frequency domain and keeps only the lowest $K$ frequencies of each hidden layer, discarding the rest. That rule assumes the low, high-energy frequencies are the ones that matter. Work on feature redundancy in other fields says this assumption can fail badly. In deep image features, the high-variance directions mostly encode background, the useful directions are scattered across the spectrum, and adding unneeded directions back degrades performance after a plateau (Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection). In operator learning the same mismatch has a concrete physical form. The solution of a fixed-frequency wave (Helmholtz) problem concentrates its energy on a ring of wavenumbers near the forcing frequency. Forced turbulence carries energy at the forcing scale. A low-pass box therefore spends parameters on modes that do not matter and can miss the ones that do. When only tens of simulations are available, every wasted spectral weight is a parameter fitted to noise.

This note proposes estimating which frequencies the output actually depends on, directly from the few training pairs and before training. The estimate uses closed-form input–output cross-spectral statistics: how much output energy at each frequency is linearly predictable from the input. The operator is then trained only on a non-contiguous set of modes selected by that score. The approach is an adjoint-free, output-aware replacement for frequency truncation. It is also cheap enough that its value in the 16–256 sample regime can be tested cleanly.

Core Claim

In low-data operator learning, the retained spectral modes should be ranked by explained output energy rather than by frequency index or input energy. A cross-spectral estimate of explained output energy is reliable at sample sizes far smaller than those needed to learn the spectral weights themselves. The prediction is sharp. Gains should appear on problems whose output spectrum is not low-pass: fixed-frequency Helmholtz, forced Kolmogorov flow, and anisotropic diffusion. On problems where low frequencies dominate, such as Darcy and Poisson, the method should reduce to ordinary truncation and match standard FNO. If gains appear uniformly or not at all, the claim is wrong.

For the simplest case, a linear translation-invariant operator in which frequencies do not mix, the score for mode $k$ is the explained output energy

$$ s_k = \frac{\left|\sum_{i=1}^{N} \hat y_i(k)\, \overline{\hat a_i(k)}\right|^2}{\sum_{i=1}^{N} |\hat a_i(k)|^2}, $$

where $\hat a_i(k)$ and $\hat y_i(k)$ are the Fourier coefficients of the $i$-th input and output fields at mode $k$, and $N$ is the number of training pairs. The score equals the output energy at $k$ captured by the best per-mode linear transfer function. Each training pair contributes one independent estimate per mode, so the relative error of $s_k$ shrinks roughly as $1/\sqrt{N}$ by the usual sample-mean argument. Ranking modes therefore needs only enough samples to resolve gaps between scores. By contrast, fitting the FNO weights at those modes requires on the order of $K d_v^2$ parameters, where $d_v$ is the hidden channel width.

For operators that couple frequencies, such as variable coefficients or nonlinear terms, we replace the per-mode ratio with a reduced-rank ridge regression from all input Fourier coefficients to each output mode, and score each mode by its explained output energy. This is the cross-covariance analogue of an active subspace.

Relation to Prior Work

Three papers motivate the proposal and define its boundary.

Derivative-enhanced Deep Operator Network projects the input onto directions found by an active subspace, the dominant eigenvectors of the averaged outer product of output gradients. It shows these directions beat variance-based Karhunen–Loève projection on nonlinear Navier–Stokes, with 16 samples outperforming vanilla DeepONet trained on hundreds. This is the strongest evidence that output-aware selection beats variance-based selection. However, that method needs adjoint solves and acts only on the input. Our score needs no derivatives and acts on the operator's own spectral modes.

Incremental Fourier Neural Operator grows the number of modes during training using the norm of the learned spectral weights. It reports 10% lower test error on Kolmogorov flow with 20% fewer modes, with the advantage largest in low-data regimes. Two features separate it from our method. It still adds modes in frequency order, and it decides using weights learned from scarce data. We select non-contiguous modes from data statistics before any weight is fitted.

Information Shapes Koopman Representation fights redundancy and collapse in finite Koopman latent spaces with an entropy regularizer. We do not regularize toward diversity. We remove capacity the output does not use, which is closer in spirit to the greedy, non-consecutive component selection of the anomaly-detection work.

Two further papers constrain the design. Priors for Diversity in Generative Latent Variable Models finds that redundant latent capacity hurts most at small sample sizes, which tells us where to look for an effect. Neural-HSS: Hierarchical Semi-Separable Neural PDE Solver shows that building known operator structure into the architecture yields large sample-efficiency gains on elliptic problems: 16 samples match FNO at 64 on 3D Poisson. That result implies our method should not be expected to beat structural priors where those priors apply. It should be compared against them honestly.

Research Program

We first build the scoring step: the per-mode estimate above, and the reduced-rank version for coupled operators, with rank and ridge strength chosen by leave-one-out on the training pairs.

Next comes the selection step. Modes are ranked by score, and the number retained is chosen by greedy forward addition. The criterion is leave-one-out error of the closed-form linear model, which aims for the start of the plateau that the anomaly-detection work observed.

The operator is a standard FNO with the same mode mask in every layer. Nonlinear layers create harmonics that deeper layers may need, so a second variant augments the mask in deeper layers with sums of selected wavenumbers. The ablations will show whether this matters.

For nonlinear problems, a refinement step trains briefly and then recomputes the score using the training residual as the target. Modes are added only where the residual still has explained energy, and the step is repeated at most twice.

Finally, an analysis component measures how stable the ranking is as a function of $N$, compared with how stable learned spectral weights are. This isolates the claim that ranking is easier than fitting.

Baselines

The naive reference is a standard FNO (Fourier Neural Operator for Parametric Partial Differential Equations) with the default contiguous $K$ and width. A second reference is the same model with $K$ chosen by cross-validation on the training set. Every method receives the same training pairs, the same width and depth, and the same hyperparameter search budget (equal numbers of trials). Model selection uses only training folds, never the test set, and results are reported over five seeds.

The strongest existing method is iFNO (Incremental Fourier Neural Operator), run with its published growth schedule and its threshold tuned within the shared budget. The file reports its gain over FNO on Kolmogorov flow only in relative terms (10% lower test error, 20% fewer modes). There is no reported result found on Lacuna for the absolute relative L2 error of FNO or iFNO on Kolmogorov flow, so we will reproduce both. Beating iFNO means lower test relative L2 error at matched or smaller mode count, at sample sizes of 64 or fewer, on at least two of the three non-low-pass problems.

A group-lasso FNO is a second strong competitor. It starts from a large contiguous $K$ and adds a group penalty on each mode's weights, with the penalty strength tuned within the budget. It is the obvious learned alternative to our pre-training selection.

On elliptic problems we also run Neural-HSS. There the file reports 16 samples matching FNO at 64 on 3D Poisson, and 2D Poisson error of $7\times10^{-8}$ against FNO's $7\times10^{-6}$. We expect Neural-HSS to win on those problems and report it as such.

The ablation isolating the key ingredient keeps the mask size fixed and changes only the ranking criterion. The three versions are explained output energy (ours), input energy (a variance-ranked, non-contiguous Karhunen–Loève analogue), and a random non-contiguous set. An oracle mask computed from the full dataset separates errors in the criterion from errors in estimating it.

Why not just…?

Why not just tune $K$ by cross-validation? With few samples a contiguous box must grow to the outermost relevant frequency to cover it, so on Helmholtz it must include every interior mode inside the ring. Cross-validation cannot fix the shape of the box, only its size. The cross-validated-$K$ baseline measures this directly. If it matches our method on Helmholtz, the shape argument fails.

Why not just use iFNO or a group lasso with large $K$? This is the real threat. A sparsity penalty can in principle discover the same non-contiguous set. The difference is that it must fit a large-$K$ model to tiny data in order to decide what to prune, and its penalty strength must be tuned on that same tiny data. Our score is closed-form and its error falls as $1/\sqrt{N}$ per mode. The ranking-stability analysis and the head-to-head comparison at $N=16$ and $N=32$ settle the question. If group lasso matches us at every $N$, the contribution reduces to a cheaper selection rule plus an analysis, and we will say so.

Why not just reduce the input with Karhunen–Loève or PCA? Derivative-enhanced DeepONet already shows variance-based projection falls well short of output-aware projection on nonlinear flow (Derivative-enhanced Deep Operator Network), and the anomaly-detection study shows the same failure in deep features. Our input-energy ablation tests this inside the FNO itself.

Why not just narrow the network or pretrain? A narrower width cuts parameters uniformly and does not move capacity toward the relevant frequencies. We include width-matched FNOs so that equal parameter counts are compared. Pretraining approaches such as DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training and Strategies for Pretraining Neural Operators (about 50% gains at 100 downstream samples) are complementary rather than competing. As a secondary experiment we apply the mask during fine-tuning of a pretrained FNO.

Proposed Experiments

The test suite has three non-low-pass problems and two controls. The non-low-pass problems are 2D Helmholtz at fixed frequency with random sources, generated by us with a finite-difference solver in a homogeneous and in a mildly heterogeneous medium; Kolmogorov flow as used by iFNO; and 2D anisotropic diffusion. The controls, where low-pass truncation should already be right, are Darcy flow and 1D Burgers from PDEBench: An Extensive Benchmark for Scientific Machine Learning. Training sizes are 16, 32, 64, 128, and 256. Metrics are test relative L2 error and PDEBench's Fourier-band error (fRMSE). Band error matters because it shows whether gains come from the frequencies we predicted.

Three checks decide the outcome. The first: on Helmholtz, the advantage over cross-validated FNO and iFNO should be large at $N\le 64$ and shrink as $N$ grows, while on Darcy and Burgers the difference should stay within seed noise. The second: at a fixed mask size, explained-output-energy ranking must beat both input-energy and random rankings. Otherwise the gain comes from sparsity, not from the criterion. The third: the overlap between the mask estimated at small $N$ and the oracle mask should exceed the overlap between the top-$K$ learned-weight sets at the same $N$, which would confirm that ranking modes is easier than fitting their weights.

A negative result is also informative. Failure on Kolmogorov flow but success on Helmholtz would show that first-order cross-spectral statistics miss modes created by nonlinear interaction. That would make residual-based refinement the necessary part of the method, not an optional one.