A physics-informed neural network (PINN) is trained on a loss that adds several terms: how badly it violates the differential equation inside the domain, how badly it misses the boundary and initial conditions, and, when measurements exist, how far it sits from the data. Practitioners tune the relative weights of these terms by hand or by expensive black-box search. Separately, the best-performing small PINNs now send their inputs through hundreds or thousands of Fourier features (sine and cosine functions of the input at many frequencies) and fit a linear layer on top. Such bases are highly redundant. This note argues that these two nuisances are one problem. Whether a Fourier feature is redundant depends on the loss weights, because redundancy has to be judged after the differential operator and the boundary map are applied to the features, not in the raw feature space. Conversely, when the top layer is linear, the whole path of loss weights can be scanned in closed form at almost no cost.
The concrete proposal is a procedure for linear-readout Fourier-feature PINNs. For each loss component, it precomputes one small matrix that summarizes how the features behave under that component's operator. It then searches loss weights, the regularization strength, and the frequency scale by solving a cheap linear system per candidate. Within that search, it prunes features with a column-selection rule applied to the weighted, operator-transformed system. If the idea holds, it should match the accuracy of tuned Fourier-feature PINNs on high-frequency problems with a fraction of the features, and at a small fraction of the tuning cost of black-box search.
Core Claim
Take features $\phi_1,\dots,\phi_m$ and a prediction $u(x)=\sum_j \theta_j \phi_j(x)$. For a linear PDE, each loss component $k$ (interior residual, boundary, initial, data) has a linear operator $\mathcal{L}_k$ evaluated at $n_k$ points. This gives a matrix $B_k\in\mathbb{R}^{n_k\times m}$ with entries $(\mathcal{L}_k\phi_j)(x_i)$ and a target vector $b_k$. With loss weights $w_k$ and ridge strength $\lambda$, the readout is
$$ \theta(w,\lambda)=\Big(\sum_k \tfrac{w_k}{n_k} B_k^\top B_k+\lambda I\Big)^{-1}\sum_k \tfrac{w_k}{n_k} B_k^\top b_k . $$
The per-component Gram matrices $B_k^\top B_k$ are $m\times m$ and are computed once. Every new $(w,\lambda)$ then costs one $m\times m$ solve instead of a training run. A feature is redundant when its column in the stacked, weight-scaled matrix $[\sqrt{w_k/n_k}\,B_k]_k$ lies nearly in the span of the other columns.
Two features can be nearly identical as functions yet very different after a second derivative is applied. Two features can also be distinct in the interior yet indistinguishable on the boundary. It follows that pruning in the raw feature space, by PCA or by feature correlation, selects the wrong subset. It also follows that the right subset shifts as the weights shift. The testable claims have three parts. First, operator-aware, weight-aware selection beats raw-space pruning at equal feature budget. Second, the selected subset measurably changes with $w$. Third, a closed-form weight search with a held-out residual criterion finds configurations as good as full black-box hyperparameter optimization at a tiny fraction of the cost. If the second claim fails, meaning the subset is stable across weights, the coupling is illusory. The method then reduces to two separate, simpler procedures, and we would report that.
Relation to Prior Work
The motivating evidence comes from two Fourier-feature PINN lines. Enhancing Physics-Informed Neural Networks Through Feature Engineering (SAFE-NET) uses a single hidden layer of 50 neurons over an adaptive Fourier basis. It reduces top Hessian eigenvalues by a factor of $10^4$ on the wave equation. Its authors concede that the needed number of features grows badly with dimension and that no scheme tunes frequencies or loss weights. Iterative Training of Physics-Informed Neural Networks with Fourier-enhanced Features (IFeF-PINN) solves a bilevel problem: gradient steps on the feature network, and a closed-form linear regression for the readout. It reports heavy memory use and strong sensitivity to the Fourier sampling scale $\sigma$. We reuse IFeF-PINN's key structural fact, that the readout is a convex least-squares problem for linear PDEs. Neither SAFE-NET nor IFeF-PINN exploits that fact to tune loss weights or to prune features, and that exploitation is exactly what we add.
The redundancy literature constrains which pruning criterion is allowed. Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection shows that in deep features, high-variance principal components often carry background structure, and that performance-driven selection of a few non-consecutive components beats both PCA and low-variance PCA. We take this as a warning against variance-based pruning. We keep its lesson, that selection must be judged by the task, and replace its labeled score with an operator-aware criterion that needs no reference solution. Priors for Diversity in Generative Latent Variable Models uses determinant-based repulsion to stop latent components from encoding the same pattern. Our column-selection rule is the deterministic, greedy relative of that idea: pivoted QR approximately maximizes the determinant of the selected Gram block. It is applied here to operator-transformed columns rather than latent vectors, and it avoids the cubic cost of sampling from a determinantal point process over large pools. Multi-label Feature Selection via Global Relevance and Redundancy Optimization solves relevance and redundancy jointly in a global matrix problem for tabular features. We share its dislike of purely greedy one-at-a-time selection, but our relevance signal is the multi-term physics loss, not label correlation.
On the hyperparameter side, Dynamic Hyperparameter Importance for Efficient Multi-Objective Optimization (HPI-ParEGO) prunes unimportant hyperparameters during Bayesian search. It has only been tested on synthetic and tabular benchmarks, and each evaluation still means a full training run. A Hessian-informed hyperparameter optimization for differential learning rate (Hi-DLR) uses curvature probes to set per-group learning rates, with hand-chosen groups. Our proposal differs from both in the same way: it replaces the black-box inner loop with an exact linear solve wherever the structure permits.
PINNACLE: PINN Adaptive ColLocation and Experimental points selection is the nearest conceptual cousin. It removes redundancy jointly across loss components, but in the set of training points, using neural tangent kernel eigenfunctions. We remove redundancy in the feature set. The two are complementary, and we treat PINNACLE's point selection as an optional front end rather than a competitor. Augmenting a Physics-Informed Neural Network for the 2D Burgers Equation by Addition of Solution Data Points shows why the weights matter. Purely data-fitted models can score lower pointwise error yet violate the equation, so the data-versus-physics balance changes the character of the solution.
Research Program
The first stage builds the closed-form machinery for linear PDEs. We draw a large pool of random Fourier features at several scales $\sigma$. We compute each component's operator matrix by automatic differentiation of the features, and store the per-component Gram matrices. We then search $(w,\lambda,\sigma)$ against a held-out criterion. When measurements exist, as in data-enabled settings, the held-out criterion is data misfit on withheld measurements. For pure forward problems, it is the sum of each component's residual on fresh held-out points, normalized by that component's residual under uniform weights.
Because the true objective is solution error, which is not available in practice, the second stage checks whether this proxy ranks configurations correctly. For this we borrow the sensitivity measures of Quantifying the Hyperparameter Sensitivity of Neural Networks for Character-level Sequence-to-Sequence Tasks, namely Spearman rank correlation and the percentage of best-equivalent configurations, computed between proxy score and reference error.
The third stage adds pruning. Inside the weight search, we run greedy pivoted column selection on the weighted stacked operator matrix down to a budget $m' \ll m$. We record how much the selected sets overlap as $w$ varies, which is the direct test of the coupling claim. The fourth stage extends to nonlinear PDEs such as viscous Burgers by Gauss–Newton linearization: each outer iteration is a weighted linear least-squares problem, and selection is repeated with a warm start. The same stage places the procedure inside IFeF-PINN's bilevel loop, so that the upper level trains features and the lower level becomes our weight-and-feature step.
Baselines
The naive reference point is a Fourier-feature PINN with the full random feature pool, uniform loss weights, a fixed ridge value, and $\sigma$ set by the default rule in each source paper. This tells us how much any tuning or pruning is worth.
The strongest existing method on the high-frequency question is IFeF-PINN with its authors' tuned settings. On the 2D Helmholtz equation with $a_1=a_2=100$, where vanilla PINNs and PINNsformer fail to converge, its primal-dual variant reports a relative $L^2$ error of 0.92% (Iterative Training of Physics-Informed Neural Networks with Fourier-enhanced Features). We will rerun it on identical collocation, boundary, and data points with the same wall-clock budget. Beating it means matching or improving 0.92% while using at most a quarter of the features and less total tuning time. Matching accuracy with far fewer features counts as a partial win.
SAFE-NET is the second strong baseline across its eight-PDE suite. The file reports rankings, first on 4 of 8 PDEs and top two on 5 of 8, but no scalar errors, so there is no reported result found on Lacuna for SAFE-NET's per-PDE error. We will rerun it and compare at equal parameter count.
The strongest tuning baseline is black-box search over the same space $(w,\lambda,\sigma,m')$ using HPI-ParEGO, together with plain random search. Each trial is a full training run, and the two are held to the same total compute as our method, tuning included. HPI-ParEGO reports surpassing standard ParEGO after about 30% of the trial budget on PyMOO. There is no reported result found on Lacuna for it on any PINN loss. Beating it means reaching its final reference error with a small fraction of its compute.
Three ablations isolate the key ingredient. The first prunes by PCA or by correlation on the raw feature matrix to the same budget $m'$. The second prunes with the operator-transformed matrix under fixed uniform weights, which isolates the role of weight-awareness. The third keeps all features with a tuned ridge penalty, which isolates the role of pruning.
Why not just…?
Why not just use ridge regularization with all features and skip pruning? This is the most serious threat. Ridge already cures the ill-conditioning that redundancy causes, and it is part of our closed-form solve anyway. If full-pool ridge matches our accuracy, the pruning component's only remaining value is memory and inference cost. The third ablation settles it directly. We will report accuracy against feature count for both methods. If ridge matches at full size, we will present pruning honestly as a compression result rather than an accuracy result. The claim we expect to survive is that at a fixed small budget, operator-aware selection clearly beats raw-space pruning. That matters because IFeF-PINN flags memory, and SAFE-NET flags feature growth with dimension, as their binding limits.
Why not just prune with PCA on the feature matrix? Gula and Bertoldo's results show that variance ranking in deep features tracks background structure rather than what the task needs. In our setting there is a sharper reason. The Helmholtz or wave operator multiplies each Fourier feature by roughly the square of its frequency, so variance in the raw space and importance under the operator can be inversely related. The first ablation tests this at matched budget. We expect the gap to be largest on the $a_1=a_2=100$ Helmholtz problem and on the multiscale convection-diffusion problem with components at $4\pi$ and $60\pi$.
Why not just tune the loss weights and $\sigma$ with standard Bayesian optimization or HPI-ParEGO? These methods work for any model, and for a nonlinear PDE with a deep feature network they may remain necessary. Their per-trial cost, however, is a full training run, whereas for a linear readout the exact answer for each weight vector costs one small linear solve. The compute-matched comparison shows whether that advantage survives once the Gram-matrix precomputation is counted. The Burgers experiments show where it erodes, since each Gauss–Newton step requires recomputation.
Why not just select the weights and the features separately? If the pruned set is essentially the same for all reasonable weights, a sequential pipeline is simpler and equivalent. The overlap measurement across the weight path is the decisive test. We will report it before claiming any benefit from doing the two jointly.
Proposed Experiments
The linear-PDE core uses the IFeF-PINN suite: 2D Helmholtz at $a_1=a_2=100$, 1D multiscale convection-diffusion, and 1D convection. It also uses the linear members of the SAFE-NET suite, including the wave equation, where the $10^4$ Hessian reduction gives a conditioning reference. We report relative $L^2$ error, feature count, peak memory, and total wall-clock time with tuning included. We also report the condition number of the weighted Gram matrix before and after pruning.
For physically meaningful error, the PDEBench advection and diffusion-reaction tasks from PDEBench: An Extensive Benchmark for Scientific Machine Learning add conservation error, boundary error, and frequency-space error. These three metrics show directly whether the chosen weights trade boundary accuracy for interior accuracy.
The data-anchored regime uses the 2D Burgers setup with solution points from the augmentation study. It also uses the 2D IAEA two-group reactor benchmark from A Data-Enabled Physics-Informed Neural Network with Comprehensive Numerical Study on Solving Neutron Diffusion Eigenvalue Problems, where the reported reference is a relative eigenvalue error below 0.005. In both, held-out data supplies a real validation signal, and the data weight becomes a genuine search dimension. The reactor eigenproblem is handled by inverse iteration, with a weighted least-squares solve at each step.
Four checks decide the proposal. The first is whether operator-aware selection beats raw-space PCA and correlation pruning at equal budget on the high-frequency problems. The second is whether the selected feature sets change substantially along the weight path. The third is whether the held-out residual proxy ranks configurations consistently with reference error, measured by Spearman correlation well above zero across the random configurations. The fourth is whether the closed-form search reaches HPI-ParEGO's final error at a small fraction of its compute. If the first and third checks hold but the second fails, the result is still useful: a fast, validated weight tuner plus an operator-aware pruner, reported as separable tools rather than a coupled method.
References
- Enhancing Physics-Informed Neural Networks Through Feature Engineering
- Iterative Training of Physics-Informed Neural Networks with Fourier-enhanced Features
- Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection
- Priors for Diversity in Generative Latent Variable Models
- Multi-label Feature Selection via Global Relevance and Redundancy Optimization
- Dynamic Hyperparameter Importance for Efficient Multi-Objective Optimization
- A Hessian-informed hyperparameter optimization for differential learning rate
- PINNACLE: PINN Adaptive ColLocation and Experimental points selection
- Augmenting a Physics-Informed Neural Network for the 2D Burgers Equation by Addition of Solution Data Points
- Quantifying the Hyperparameter Sensitivity of Neural Networks for Character-level Sequence-to-Sequence Tasks
- PDEBench: An Extensive Benchmark for Scientific Machine Learning
- A Data-Enabled Physics-Informed Neural Network with Comprehensive Numerical Study on Solving Neutron Diffusion Eigenvalue Problems