Scale-Aware Multi-Fidelity Search for Cosmological Field Emulators

Neural emulators for cosmology are networks that turn the smooth early-universe matter distribution into the clumpy late-time one in under a second, a job that otherwise requires a full gravitational simulation. They are becoming the default way around the cost of those simulations: the Quijote suite alone took more than 35 million core hours. Yet the emulators themselves are designed by hand, and nobody knows whether their depth, receptive field, conditioning scheme and loss weights are anywhere near good. The obvious fix is to run an off-the-shelf automated tuner over them. We argue that this fix is subtly wrong for physics. The cheap trial runs such tuners depend on, such as shorter training or coarser grids, are informative about some physical length scales and blind to others. A tuner that treats "emulator quality" as one number will reward designs that look good on large scales and quietly discard the ones that get small-scale gravitational collapse right, which are exactly the scales where current emulators fail.

We propose a multi-fidelity Bayesian optimizer that models emulator error separately in each band of spatial frequency. A Bayesian optimizer fits a probabilistic model of "configuration → score" and uses it to choose the next trial; "multi-fidelity" means it can also pay for cheap approximate trials. Our version also learns, and partly knows in advance from physics, which cheap trial informs which band. A run trained on a coarse particle grid, for instance, cannot contain information above that grid's resolution limit. Two sentences suffice to explain the idea to a colleague. Cheap evaluations of a physics emulator are informative scale by scale, so the tuner should track error per scale and spend its budget on whichever cheap run is informative about the scales still uncertain. We will first measure whether this matters at all, by checking whether any single cheap proxy preserves the ranking of designs at small scales. Only then will we test whether exploiting the per-scale structure finds better emulators per GPU-hour than existing tuners.

Core Claim

The claim is that no single low-cost fidelity preserves the ranking of cosmological emulator designs at all spatial scales, and in particular among the best designs at small scales. The corpus contains a direct tension on this point. Rethinking Performance Estimation in Neural Architecture Search found input resolution to be among the least important proxy settings for preserving architecture rankings on image classification. In cosmology, however, resolution sets which non-linear modes exist at all, and the hand-built emulators of Field Level Neural Network Emulator for Cosmological N-body Simulations and Field-level Emulation of Cosmic Structure Formation with Cosmology and Redshift Dependence are accurate at the percent level only down to $k \sim 1\,h\,\text{Mpc}^{-1}$, where $k$ is spatial frequency. Below that scale they produce fuzzy halos. If rankings are scale-dependent, the correct object to model is a per-band error surface, and the correct acquisition rule selects a fidelity by the bands it informs.

Concretely, let $\lambda$ be an emulator configuration and $f$ a fidelity (training grid resolution, training crop size, number of training simulations, epochs). Let $y_b(\lambda, f)$ be the error in band $b$ of the matter power spectrum and cross-correlation statistics. The quantity we optimize is the full-fidelity weighted error

$$E(\lambda) = \sum_b w_b \, y_b(\lambda, f_{\max}),$$

where the weights $w_b$ are set by the science use, for example uniform over $0.05 < k < 2\,h\,\text{Mpc}^{-1}$. The surrogate is a multi-output model over $(\lambda, f, b)$ with a learned correlation $\rho_b(f)$ between each cheap fidelity and full fidelity in band $b$. It carries a hard prior that $\rho_b(f)=0$ for bands above the resolution limit of fidelity $f$. Each step chooses the trial $(\lambda, f)$ that maximizes the expected reduction in uncertainty about the minimizer of $E$, divided by the GPU cost of $(\lambda, f)$.

Research Program

The work has two stages, and the first stage can falsify the premise of the second.

Stage one is a rank-preservation atlas. We define a search space around the styled 3D convolutional emulator of Field-level Emulation of Cosmic Structure Formation with Cosmology and Redshift Dependence. Its dimensions are the number of downsampling levels (which controls receptive field), channel width, kernel size, which layers receive the cosmology-dependent style modulation, the relative weights of the Lagrangian displacement, Eulerian density and velocity loss terms, learning rate and weight decay. We sample roughly 100 configurations and train each at full fidelity on Quijote. Each is also trained at a grid of cheap fidelities: 256³-derived inputs versus 512³, 64³ versus 128³ crops, 10% versus 100% of the training simulations, and 5% versus 100% of epochs. For every (fidelity, band) pair we report Kendall's $\tau$ against the full-fidelity ranking, both over all configurations and over the top decile. The top-decile view matters because EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring documents that ranking proxies degrade specifically among the best candidates. The atlas is useful by itself, since it would be the first map of which cheap experiments can be trusted for 3D physics emulators. It also directly supplies $\rho_b(f)$ estimates to warm-start stage two.

Stage two builds the scale-aware optimizer. It reuses the successive-halving scheduling of BOHB: Robust and Efficient Hyperparameter Optimization at Scale for its robustness and asynchronous parallelism. We replace BOHB's single-score density model with the per-band multi-output surrogate and add the band-aware, cost-normalized acquisition rule. As a secondary scientific question, we examine whether the configurations the optimizer selects are compact rather than large. That would reproduce, in a 3D setting, the "bigger is not better" effect that Auto-PINN: Understanding and Optimizing Physics-Informed Neural Architecture found for small differential-equation solvers. Auto-PINN searched sequentially and by hand-ordered stages on 1D and 2D problems. We use a formal multi-fidelity search and physical fidelities.

A cheap testbed comes first. This is the FastPM setting of Learning to Predict the Cosmological Structure Formation, with 32³ particles and 10,000 training pairs. There, full-fidelity ground truth for every configuration is affordable, so optimizer regret can be computed exactly. Afterwards we move to Quijote from The Quijote simulations, using its roughly 2,000-simulation Latin-hypercube split with extreme cosmologies held out, as in the 2024 emulator paper.

Baselines

The naive reference point is the published hand-designed emulator trained as described by its authors, with no search. Field Level Neural Network Emulator for Cosmological N-body Simulations reports percent-level power spectrum accuracy down to $k \sim 1\,h\,\text{Mpc}^{-1}$, stochasticity below 2% for most cosmologies, and about 10% accuracy on halo numbers and profiles. On the FastPM testbed the corresponding reference is D³M from Learning to Predict the Cosmological Structure Formation, which reports 2.8% average relative displacement error against 9.3% for second-order perturbation theory. We retrain both with the authors' settings on our splits, so that differences come from design rather than data. Any searched model must beat these on held-out cosmologies in the $1 < k < 2$ band without losing accuracy at large scales.

The strongest existing method is BOHB run with its standard fidelity, training epochs, over the same search space, the same Quijote data and the same total GPU-hour budget. It is the strongest competitor because it already handles mixed architecture and training settings with multi-fidelity pruning. Its own paper reports speedups of 20× to 55× over baselines and up to 100× over Hyperband on its benchmarks. We found no reported result on Lacuna for BOHB, or any automated tuner, on cosmological field emulators. A second strong competitor is BOHB given our physical fidelities but still optimizing a single aggregated score. We also run Budgeted Performance Estimation from Rethinking Performance Estimation in Neural Architecture Search: it chooses one cheap proxy setting by rank correlation and then runs evolutionary search under it. That paper reports cutting search from 3,150 to 0.33 GPU days on CIFAR-10. Random search at equal budget is the floor.

Two ablations isolate our ingredients. The first keeps physical fidelities but collapses the per-band surrogate to a scalar. It tests whether per-scale modeling, not merely new fidelities, is what helps. The second drops the hard resolution-limit prior and lets $\rho_b(f)$ be fully learned. It tests whether the physics prior earns its place. "Beating" a baseline means lower full-fidelity weighted error $E$ at matched GPU-hours, with the gap exceeding the seed-to-seed spread over five optimizer seeds. It also means reaching the baseline's final $E$ with at least 2× less compute.

Why not just…?

Why not just run BOHB with epochs as the fidelity? This is the most natural thing to do, and it may work well on large scales. Epoch-truncated training is known to preserve local but not global rankings, as Rethinking Performance Estimation in Neural Architecture Search documents. Small-scale accuracy in these emulators is plausibly a late-training phenomenon, which would make early pruning discard exactly the designs we want. Stage one measures this directly through the top-decile $\tau$ of short-epoch runs in high-$k$ bands. If that $\tau$ is high, plain BOHB is adequate and we will say so.

Why not just pick one good proxy once, BPE-style, and search under it? This is the real threat. If some single cheap setting, such as full resolution with few simulations, preserves top-decile rankings in every band, then per-band modeling is unnecessary overhead. The atlas settles this. Our method is warranted only if the best single fidelity's top-decile $\tau$ in the smallest-scale bands is clearly below its $\tau$ at large scales, and the fidelities that best predict different bands differ. If not, we will report BPE with a physics-validated proxy as the recommended practice and drop stage two's claim.

Why not just scale up the hand-designed U-Net and sweep the learning rate? Capacity is cheap to add, and it could fix fuzzy halos. However, Auto-PINN: Understanding and Optimizing Physics-Informed Neural Architecture found that larger physics networks can train less stably and end up less accurate. The fuzzy-halo failure in Field-level Emulation of Cosmic Structure Formation with Cosmology and Redshift Dependence may instead depend on receptive field or on how the loss terms are weighted. We include a width-and-depth scaling series with a learning-rate sweep, at matched compute, as an extra baseline. If it matches the searched model at small scales, the answer is "just scale."

Why not just use zero-cost proxies and skip training altogether? Zero-cost proxies score an untrained network from a single forward or backward pass. The best evolved proxy reaches $\tau = 0.65$ on NAS-Bench-201 in EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring. That proxy was built for 2D convolution–batch-norm–ReLU cells, degrades among top candidates, and knows nothing of per-scale error. We compute Synflow, NASWOT and EZNAS-A on the stage-one configurations as a cheap check, but we expect low high-$k$ correlation.

Proposed Experiments

The FastPM experiment comes first. We will enumerate a reduced search space fully at full fidelity so that exact regret curves exist for every method. The decisive checks are three. The first is the stage-one atlas on Quijote, where the per-band, top-decile Kendall $\tau$ matrix decides whether stage two is justified. The second is equal-budget regret on $E$, together with per-band error curves on held-out extreme cosmologies, for our method against BOHB with epochs, BOHB with physical fidelities and a scalar score, BPE plus evolution, the scaling series, and random search. The third is a downstream test of the best configuration from each method, scored on bispectrum residuals, the halo mass function and stochasticity. These are statistics that were not part of $E$, so the test shows the searched gains are not overfit to the power spectrum.

A negative result at the first check would still be informative. It would tell emulator builders that one cheap proxy can be trusted, and it would name that proxy.