Cosmologists want to know whether the primordial density fluctuations left by inflation were perfectly Gaussian or carried a faint "three-point" signature, called primordial non-Gaussianity. The shape of that signature tells apart competing theories of the early universe. Neural networks trained on simulated universes can read the local shape of this signal well, but they fail on the equilateral shape. That failure is puzzling, because a hand-built Fisher forecast says the same topological summaries of the cosmic web contain enough information to constrain it. We propose a specific, testable explanation. When the signal is weak, the regression target behaves like heavily noisy labels. The networks were then probably sized near their interpolation threshold, the capacity at which a model can just barely fit its training set. In deep learning that is exactly where test error and input sensitivity both peak, the "double descent" bump.
The practical consequence is a cheap way to choose architectures for scientific inference. We would score each candidate network by the Fisher information its output carries about the cosmological parameters. Fisher information here is the curvature of the likelihood, and its inverse gives the smallest achievable error bar. This score needs only forward passes on simulations that already exist. Classification-flavored zero-cost proxies, pruning-style scores computed at random initialization, would not be used for ranking. We would then use the measured double-descent curve of the network's input sensitivity to decide which widths are worth searching at all. The claim is that, at fixed simulation budget, the error bar on the equilateral amplitude from a neural summary depends non-monotonically on width. It should peak where the network's empirical Lipschitz constant peaks, and it should approach the hand-built Fisher bound only once the network is well past that point.
Core Claim
The empirical Lipschitz constant is the largest factor by which the network amplifies a small change in its input. Several studies track it across width and find that it follows the double-descent shape of test error: it rises to a peak at the interpolation threshold and falls after. Some Fundamental Aspects about Lipschitz Continuity of Neural Networks measures a Jacobian-based lower bound and shows its peak aligns with the test-loss peak on CIFAR-100. Deep Double Descent via Smooth Interpolation and Are All Linear Regions Created Equal? find the same pattern in input-space sharpness and in the effective nonlinearity of ReLU networks. All of this work is on classification and translation benchmarks. None of it asks what the peak does to a scientific estimator whose job is a small error bar.
Our first contribution is to connect the two. Let $s(x)$ be the network output on a persistence image $x$, a pixelated summary of when topological features of the halo distribution appear and disappear. Let $\mu(\theta)$ be the mean of $s$ over simulations at parameters $\theta$, and let $\Sigma$ be its covariance over fiducial realizations. The Fisher information of the learned summary is then
$$F_{ij} = \partial_i \mu^{\top}\, \Sigma^{-1}\, \partial_j \mu .$$
To first order, $\Sigma \approx \mathbb{E}[J]\, C_x\, \mathbb{E}[J]^{\top}$, where $J$ is the input-output Jacobian and $C_x$ is the cosmic-variance covariance of the input. Similarly, $\partial_i \mu \approx \mathbb{E}[J]\,\partial_i \bar{x}$, where $\partial_i \bar{x}$ is the shift in the mean image as parameter $i$ changes. A Jacobian that grows large in directions dominated by cosmic variance, which is what an interpolation-threshold network does when it memorizes realization-specific fluctuations, inflates $\Sigma$ without increasing the signal. It therefore shrinks $F$.
For the equilateral amplitude the per-simulation signal-to-noise is low. The target is close to a noisy label, and noisy labels are the regime where the double-descent peak is sharpest. We therefore predict that the constraint on the equilateral amplitude from the learned summary, $\sigma(f_{\rm NL}^{\rm equi}) = (F^{-1})^{1/2}_{\rm equi,equi}$, gets worse at the threshold width and recovers beyond it. We also predict that the local amplitude, which has more signal per realization, shows a much weaker bump.
Research Program
We build on the likelihood-free pipeline of Cosmology with Persistent Homology: Parameter Inference via Machine Learning. That work trains fixed, heuristically chosen convolutional networks on persistence images from QUIJOTE-PNG to predict posterior means and variances. It reports that these networks beat power spectrum plus bispectrum for $\Omega_m$, $\sigma_8$, $n_s$ and $f_{\rm NL}^{\rm loc}$, but struggle on $f_{\rm NL}^{\rm equi}$. That finding motivates the project. We keep its data and targets and change three things.
First, we map the double-descent curve on this task directly. At several fixed training-set sizes drawn from the QUIJOTE-PNG Latin-hypercube sets, we train persistence-image networks across a sweep of width multipliers. For each network we record test loss, $R^2$ per parameter, and the Jacobian lower bound $C_{\rm lower} = \max_x \|J(x)\|_2$. The last is computed as in Some Fundamental Aspects about Lipschitz Continuity of Neural Networks; we do not use the layer-norm product, which that paper shows reaches $10^{40}$ in ResNet50 and is useless.
Second, for every trained network we compute the summary Fisher matrix above. The covariance $\Sigma$ comes from the 15,000 QUIJOTE-PNG fiducial realizations. The derivatives $\partial_i \mu$ come from the paired one-parameter-at-a-time "Fisher runs" that The Quijote simulations built for exactly this purpose. The large fiducial set keeps the bias from inverting a noisy covariance small, since the summary has only five dimensions. We check how stable the derivatives are as the number of derivative simulations changes, because finite-difference noise is a known weakness of simulation Fisher matrices.
Third, we need a reference for "all the information there is." Fisher Forecasts for Primordial non-Gaussianity from Persistent Homology computes a linear Fisher forecast on 1D histograms of topological features from the Eos simulations. Because Eos differs from QUIJOTE-PNG in halo selection and simulation details, we recompute that histogram Fisher forecast on the same QUIJOTE-PNG halo catalogs and filtration. That makes the neural and hand-built summaries directly comparable.
The architecture-search layer is deliberately modest. We define a small cell-based search space over persistence-image networks in the style of NAS-Bench-201, crossed with width multipliers, sized so that an exhaustive ground truth can be trained once. This extends NAS to a non-vision, topological input, a domain that benchmark has never covered. We then ask how cheaply the final $\sigma(f_{\rm NL}^{\rm equi})$ ranking can be recovered. The candidate cheap scores are the summary Fisher after a short training run, and a width filter that discards candidates whose short-run $C_{\rm lower}$ trajectory places them near the threshold. We use trained or partially trained Lipschitz values, not values at initialization. The double-descent phenomenon is a property of trained networks, and nothing in the file suggests the initialization-time Jacobian shows the peak.
Baselines
The naive reference point is the published setup: the fixed convolutional backbone of Cosmology with Persistent Homology: Parameter Inference via Machine Learning, trained with its recipe on the same splits. The file gives no per-parameter $R^2$ values for it, so we will measure them ourselves (no reported result found on Lacuna for the persistence-image CNN $R^2$ on $f_{\rm NL}^{\rm equi}$). Beating it means a lower Fisher error bar and a higher held-out $R^2$ on $f_{\rm NL}^{\rm equi}$ at the same training-set size.
The strongest competitor on the question of information content is the linear Fisher forecast on persistent-homology histograms from Fisher Forecasts for Primordial non-Gaussianity from Persistent Homology. It reports marginalized $\Delta f_{\rm NL}^{\rm loc} \sim 16$ and $\Delta f_{\rm NL}^{\rm equi} \sim 41$ for a $1\,(h^{-1}{\rm Gpc})^3$ redshift-space volume on Eos. We rerun it on our QUIJOTE-PNG catalogs, using the same simulations for its covariance and derivatives as for the neural summaries. The past-threshold network counts as beating it if its summary Fisher error on $f_{\rm NL}^{\rm equi}$ is at or below the recomputed histogram value. We also require that the error is confirmed by the empirical scatter of the network's predictions on held-out fiducial simulations, not only by the Fisher estimate, which assumes the summary is Gaussian. Matching the bound is itself a positive result, because it would mean the ML failure was fully an optimization and sizing artifact.
Gradient-boosted trees on persistence histograms are the strongest cheap estimator. The persistent-homology ML paper found them comparable to neural networks, especially for $f_{\rm NL}^{\rm loc}$. We tune them with the same hyperparameter budget and training sets. Their summary Fisher is computed identically, by treating tree predictions as $s(x)$ with finite-difference derivatives.
For architecture ranking, the baseline is the zero-cost proxy ensemble from Zero-Cost Proxies for Lightweight NAS. It reports a Spearman rank correlation of 0.82 on NAS-Bench-201 for a Synflow, Jacob_cov and SNIP vote, against 0.61 for the reduced-training proxy EcoNAS. We also run the individual proxies from NAS-Bench-Suite-Zero. That paper warns that these proxies track parameter count, so we include parameter count itself as a baseline. Our score beats them if its Spearman correlation with the fully trained $\sigma(f_{\rm NL}^{\rm equi})$ ranking is higher at equal wall-clock budget.
Two ablations isolate the key ingredient. The first removes the width filter, so short-run Fisher scoring is used alone. The second replaces the summary Fisher with short-run validation mean-squared error at the same training cost. This tests whether the physically meaningful score matters or whether any short-training signal does equally well.
Why not just…?
Why not just use gradient-boosted trees on the histograms? This is a real threat for $f_{\rm NL}^{\rm loc}$, where they already match neural networks. The same paper reports that tree and MLP models also struggle with $f_{\rm NL}^{\rm equi}$. Trees work on 1D marginals, while the full persistence image keeps the joint birth-death structure, which may carry the equilateral information. If tuned trees reach the recomputed histogram Fisher bound on $f_{\rm NL}^{\rm equi}$, the image-network part of this proposal is unnecessary, and we will report that.
Why not just tune weight decay or early stopping, or train much wider? This is the most serious alternative, because regularization can flatten a double-descent peak. It does not compete with the hypothesis, though; it follows from it. If early stopping alone recovers $f_{\rm NL}^{\rm equi}$, that confirms the failure was a sizing and fitting artifact rather than missing information. The width sweep is run both with and without tuned regularization so this is settled directly. What remains of the proposal in that case is a cheap physical score that tells practitioners when they are in the bad regime, without training every candidate to convergence.
Why not just train the network to maximize the Fisher information of its output directly? This is a reasonable alternative objective, and we include it as an extra arm on the best architecture. It changes the loss, not the capacity question. A Fisher-maximizing network can still memorize cosmic variance near its interpolation threshold, and the width sweep with this loss tests whether the bump persists.
Why not just use the standard zero-cost proxies? They were validated on image classification. NAS-Bench-Suite-Zero shows they are biased toward large models. Our hypothesis says bigger is not monotonically better near the threshold, so a size-biased proxy should mis-rank exactly the candidates that matter. The ranking comparison above measures this directly.
Proposed Experiments
The decisive experiment is the width sweep at three training-set sizes. The hypothesis predicts three things together. The peak in $C_{\rm lower}$, the peak in test loss, and the peak in $\sigma(f_{\rm NL}^{\rm equi})$ from the learned summary should coincide in width. The coincident peak should move to larger width as the training set grows, which is the signature of an interpolation-threshold effect rather than a fixed architectural limit. And the bump should be markedly weaker for $f_{\rm NL}^{\rm loc}$.
We also test whether the first-order argument actually operates. On held-out fiducial simulations, we check whether the growth of $\Sigma$ at the peak is explained by the Jacobian's alignment with the cosmic-variance directions of $C_x$.
A second check compares the best past-threshold summary against the recomputed histogram bound. We also examine the "confidently wrong" regime noted in Deep Double Descent via Smooth Interpolation. The concern is that very wide networks might smooth toward the prior mean of $f_{\rm NL}^{\rm equi}$, giving small scatter but biased predictions. We test for this by fitting the slope of predicted against true $f_{\rm NL}^{\rm equi}$ on held-out Latin-hypercube simulations.
The ranking experiment trains the full search space once. It reports Spearman correlation and GPU-hours for each cheap score against the zero-cost ensemble and against parameter count.
The idea is falsified if $\sigma(f_{\rm NL}^{\rm equi})$ is flat across width and never approaches the histogram bound. That would mean the persistence-image representation, or the Gaussian-summary assumption behind the forecast, overstates the available equilateral information. That outcome is still informative for how much to trust topological Fisher forecasts.
References
- Some Fundamental Aspects about Lipschitz Continuity of Neural Networks
- Deep Double Descent via Smooth Interpolation
- Are All Linear Regions Created Equal?
- Cosmology with Persistent Homology: Parameter Inference via Machine Learning
- The Quijote simulations
- Fisher forecasts for primordial non-Gaussianity from persistent homology
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- Zero-Cost Proxies for Lightweight NAS
- NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies