Many scientific imaging and estimation tasks are inverse problems: we see blurred, indirect, noisy measurements and want the hidden cause behind them. Examples include conductivity inside a body from boundary voltages, or soil permeability from surface settlement. These problems are ill-posed. The physics smooths away fine detail, so a naive inversion amplifies sensor noise into wild artifacts. Neural networks trained on simulated (measurement, answer) pairs now solve such problems well. The field also assumes that generating more simulations stops helping past some point.
This note argues that part of that apparent plateau is self-inflicted. Every learned inverse solver carries a regularization knob, a single number that trades fitting the data against staying smooth or close to a prior. Practitioners tune it once at one training-set size and then hold it fixed while scaling the data. The best setting of that knob should fall as the training set grows, because a better-learned prior needs less explicit smoothing. So a fixed knob drives the solver toward over-regularization as data grows. Our claim is that this produces a false plateau, sitting above the real limit. The real limit is set by the noise level and by how fast the physics destroys information. We propose to separate the two, predict where genuine saturation begins, and give a cheap rule for retuning the knob as data scales.
Core Claim
The claim has two parts.
The first part concerns the true floor. For a learned inverse solver, the error that remains as the number of training pairs $n$ grows is set by the forward operator's information loss and the noise level $\delta$. Data volume does not set it. The point where returns diminish can be predicted from these two quantities.
The second part concerns the fixed knob. Holding the regularization weight $\lambda$ fixed across $n$ makes scaling curves flatten earlier and higher than this floor.
The prediction comes from the linear-Gaussian case, where everything is computable. Let $s_k$ be the operator's singular values, which measure how strongly it passes each mode of the unknown. Let $p_k$ be the prior variance of each mode. Then the best achievable error and the excess error from learning with $n$ samples behave as
$$ E_\infty(\delta)=\sum_k \frac{p_k\,\delta^2}{s_k^2 p_k+\delta^2},\qquad E(n)-E_\infty \approx \frac{c\, d_{\mathrm{eff}}(\delta,\lambda)}{n}. $$
Here $d_{\mathrm{eff}}$ is the number of modes the measurements still resolve above the noise, meaning roughly those with $s_k^2 p_k>\delta^2$. The excess term takes the usual $1/n$ form of regularized least squares. Its coefficient depends on $\lambda$, and so does the optimal $\lambda_n^\star$.
Two consequences can be tested. When the singular values decay exponentially (severely ill-posed problems), $d_{\mathrm{eff}}$ grows only logarithmically in $1/\delta$. Such problems should saturate at much smaller $n$ and a higher floor than mildly ill-posed, polynomially decaying ones. Raising the noise should also bring saturation earlier, because $d_{\mathrm{eff}}$ shrinks. This answers the open tension between the monotone $O(1/n)$–$O(1/\sqrt n)$ risk decay proved in On Learning the Optimal Regularization Parameter in Inverse Problems and the noise-floor plateaus reported in Physics-Informed Neural Networks for Fast 3D Consolidation Prediction and Numerical identification of a Robin coefficient in parabolic problems. On this account both are correct: the risk keeps decaying, but toward $E_\infty(\delta)$.
Relation to Prior Work
The closest work is On Learning the Optimal Regularization Parameter in Inverse Problems. It picks a scalar $\lambda$ for a classical regularizer by empirical risk minimization (choosing the value with the lowest error on paired training data) over a grid. It proves that the chosen value approaches the ideal one at $O(1/n)$ or $O(1/\sqrt n)$, with only logarithmic dependence on grid size. We reuse its selection procedure as a tool. Our question is different: it does not ask how the optimal $\lambda$ of a learned solver drifts with $n$, nor what holding $\lambda$ fixed does to measured scaling curves.
The data-saturation literature motivates the question. It includes curated 6K-sample instruction sets beating 10× larger pools in What Makes Good Data for Alignment?, and the mid-training influence "basin" in Capturing the Temporal Dependence of Training Data Influence. Both attribute plateaus to redundancy in the data. We test a competing explanation, regularization misspecification, in a setting where the true floor is computable.
Several solvers supply our testbeds, and each tunes its regularization once on a fixed dataset:
- Invertible Fourier Neural Operators (iFNO) uses static loss weights and a fixed β-VAE (variational autoencoder) weight across its training stages.
- Transformer Meets Boundary Value Inverse Problems trains at a single size of 10,800 samples. - Fast Voxel-Wise Kinetic Modeling in Dynamic PET uses fixed loss balances on 70 scans.
Work on adaptive penalties constrains the method we can propose. Proximal alternating penalty algorithms and A Trust-Region Method For Nonsmooth Nonconvex Optimization show that adaptive rules reintroduce sensitivity through initial scales. We therefore avoid rules with new free constants and extrapolate from measured points instead.
Research Program
Exact linear testbed. We first build a fully controlled testbed: linear forward operators with chosen singular-value decay, polynomial or exponential, and Gaussian priors and noise. Here $E_\infty(\delta)$, $d_{\mathrm{eff}}$, and the optimal $\lambda_n^\star$ are known in closed form or by cheap computation. We train both a regularized linear reconstructor and a small neural reconstructor at $n$ ranging over three orders of magnitude, under two protocols. In the fixed-λ protocol, λ is tuned at the smallest $n$ and frozen. In the oracle protocol, λ is retuned at every $n$ by empirical risk minimization on a held-out split. We record the gap between the two curves, the measured saturation point, and agreement with the predicted floor.
Neural benchmarks. We then carry the same two protocols to neural solvers. In each solver we identify one dominant regularization knob:
- in iFNO on Darcy D-LINE at 10% and 20% noise, the β weight;
- in the U-Integral Transformer on synthetic electrical impedance tomography, weight decay together with training-noise augmentation strength;
- in the consolidation surrogate, the physics-residual weight.
We subsample training sets geometrically. For electrical impedance tomography that runs up to the full 10,800 samples; for Darcy and consolidation we generate more simulations.
The cheap rule. Finally we propose a scale-aware rule that avoids per-$n$ grid search. It selects $\lambda$ by empirical risk minimization at two small pilot sizes. It then extrapolates $\lambda_n^\star$ to the target $n$ using a power law whose exponent is fitted on the pilots. The linear theory supplies the functional form; the data supply the exponent. A second variant is a within-run rule that uses no extra trials. During training it lowers regularization whenever the held-out forward residual sits above the noise level, following the discrepancy-principle logic of On the Discrepancy Principle for Stochastic Gradient Descent. Because it needs $\delta$, we test it with both known and estimated noise levels.
Baselines
Naive reference. The do-nothing reference is the standard practice: λ is tuned once at a single dataset size by the authors' recipe, then held fixed as data scales. For iFNO we use the released iFNO code with its default β. For the U-Integral Transformer we use its published configuration. Any plateau this protocol shows is the phenomenon under test.
Strongest existing method. The strongest baseline is per-$n$ empirical-risk-minimization selection of λ over a log grid, following On Learning the Optimal Regularization Parameter in Inverse Problems. That paper reports only rates, $O(1/n)$ or $O(1/\sqrt n)$ excess risk, and an advantage over the quasi-optimality criterion; the file gives no absolute error figure. This baseline is effectively an oracle that costs one training run per grid value at every $n$. Our extrapolation rule beats it if it matches its error within the grid resolution while using at least 5× fewer training runs. We do not expect to beat its accuracy.
Published reference numbers. On the architecture side, we compare with what the cited papers report on the same tasks:
- iFNO's relative error of about 0.11 on D-LINE at 20% noise, against about 0.17 for standard FNO (Invertible Fourier Neural Operators);
- the U-Integral Transformer's relative $L^2$ error of 0.163 and Dice of 0.897, against 0.200 and 0.845 for U-Net, with its Dice of 0.845 at 20% noise (Transformer Meets Boundary Value Inverse Problems).
A result counts as meaningful if retuning λ at full data lowers the U-Integral Transformer's or iFNO's error below these published values with an unchanged architecture. For the consolidation surrogate we use the published accuracy figures, over 95% $C_v$ accuracy at 30% noise, only as a sanity check.
Curation ablation. A third baseline isolates the redundancy explanation. We apply diversity-based subset selection (embedding-distance filtering in the style of DEITA) at fixed λ.
Fairness. All arms share the same data, architecture, total epochs, and a matched hyperparameter budget. The fixed-λ arm also receives the full grid search at its single tuning size.
Why not just…?
Why not just grid-search λ again at every dataset size? This is the correct oracle, and it is a real threat to the method part of the proposal. It is not a threat to the finding. If per-$n$ retuning removes most of the plateau, the central claim holds, and the remaining contribution is cost. The extrapolation rule matters only if it matches the oracle at a fraction of the runs. The pilot-to-target experiments settle this directly. If extrapolation fails, we will report per-$n$ retuning as the recommended practice.
Why not just curate the data, as instruction-tuning work does? What Makes Good Data for Alignment? shows that redundancy causes plateaus in language model tuning. Simulated inverse-problem datasets, however, are drawn i.i.d. from a known prior, so redundancy should be weak. The curation ablation tests this. If curation at fixed λ closes as much of the gap as retuning does, our misspecification account is wrong. Running both together tells us whether the two effects add.
Why not just use a bigger or better-designed architecture? Architecture does help: iFNO roughly 0.11 versus FNO roughly 0.17 at 20% noise. But no architecture can go below $E_\infty(\delta)$, which in the linear testbed we compute exactly. We will report where each architecture's retuned curve sits relative to that floor. If a larger model reaches the floor with fixed λ, the knob is irrelevant for that model, and we will say so.
Why not just use the discrepancy principle or L-curve classically? These rules need the noise level or behave unstably. Rodriguez et al. found that quasi-optimality underperforms empirical risk minimization. We include the discrepancy rule as our within-run variant precisely to measure its sensitivity to misestimated $\delta$.
Proposed Experiments and Decisive Checks
Linear testbed. The first check asks whether the fixed-λ curve flattens measurably above $E_\infty(\delta)$ while the retuned curve tracks the predicted $d_{\mathrm{eff}}/n$ decay down to the floor. It also asks whether the measured saturation size shrinks with noise and with faster singular-value decay as predicted, and whether $\lambda_n^\star$ decreases with $n$.
Neural benchmarks. On Darcy D-LINE at 10% and 20% noise, electrical impedance tomography at noise levels 0, 0.05, and 0.2, and 3D consolidation, the metrics are relative $L^2$ error, plus Dice for impedance tomography. We plot error against $n$ for each protocol and report the fixed-versus-retuned gap at full data. We also report the saturation size, defined as the $n$ beyond which doubling the data gives less than 5% relative improvement.
What would falsify the claim. The decisive negative result would be a fixed-versus-retuned gap within seed noise on all three neural benchmarks. That would mean neural plateaus in inverse problems really are data-intrinsic, and that one-time tuning is safe; this is worth knowing either way.
What would support the cheap rule. The decisive positive result for the method would be the two-pilot extrapolation landing within one grid step of the oracle $\lambda_n^\star$ at the largest $n$ on at least two of the three benchmarks.
References
- On learning the optimal regularization parameter in inverse problems
- Physics-Informed Neural Networks for Fast 3D Consolidation Prediction: A Surrogate Modelling Application
- Numerical identification of a Robin coefficient in parabolic problems
- What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning
- Capturing the Temporal Dependence of Training Data Influence
- Invertible Fourier Neural Operators for Tackling Both Forward and Inverse Problems
- Transformer Meets Boundary Value Inverse Problems
- Fast Voxel-Wise Kinetic Modeling in Dynamic PET using a Physics-Informed CycleGAN
- Proximal alternating penalty algorithms for nonsmooth constrained convex optimization
- A Trust-Region Method For Nonsmooth Nonconvex Optimization
- On the Discrepancy Principle for Stochastic Gradient Descent