High-dimensional integrals, meaning averages of a function over many continuous variables, sit inside Bayesian inference, option pricing and simulation-based science. Plain Monte Carlo averages random samples. Its error falls as $N^{-1/2}$ regardless of dimension, but slowly. A control variate is a second function whose average is known exactly; subtracting it from the integrand removes most of the noise. Neural networks can be built so that their average is zero by construction, which makes them control variates automatically. They have been shown to work in moderate dimension, but they are hard to train, and they degrade as dimension grows.
This note proposes training such a network the way one teaches a student. The network first sees only the single direction along which the integrand varies most. It is allowed to use the next most important directions only after it has learned to cancel variance along the first. It finishes with the full space. Because the zero-mean property holds at every stage, the final estimate is unbiased whatever schedule is used. The schedule can only change how much variance is removed per integrand evaluation. The bet is that this ordering lets ordinary first-order optimizers fit a good control variate in 50 to 100 dimensions, where training on all coordinates at once stalls.
Core Claim
Bayesian Stein Networks (Baysian numerical integration with neural networks) apply a Langevin-Stein operator to a network output $u(x)$. The operator is $\mathcal{L}_\pi u(x) = \nabla\cdot u(x) + u(x)\cdot\nabla\log\pi(x)$, where $\pi$ is the target density, known only up to a constant. The result is the model
$$g(x) = \theta_0 + \mathcal{L}_\pi u(x),$$
whose mean under $\pi$ equals the scalar $\theta_0$. Once $g$ is fitted to the integrand $f$, the estimator $\hat I = \frac{1}{N}\sum_i [f(x_i) - \mathcal{L}_\pi u(x_i)]$, computed on fresh samples $x_i\sim\pi$, is unbiased for $I=\mathbb{E}_\pi[f]$. Its variance is $\mathrm{Var}_\pi[f - \mathcal{L}_\pi u]/N$.
That paper reports two difficulties. The loss landscape is narrow enough that Adam fails and L-BFGS is needed. And it evaluates only up to $d=20$ on Genz functions. Separately, Conditional Quasi-Monte Carlo with Constrained Active Subspaces shows that realistic integrands have a few dominant directions. These are the leading eigenvectors $U$ of $C=\mathbb{E}[\nabla f\,\nabla f^\top]$, the average outer product of the integrand's gradient.
We combine the two. At stage $k$, the vector field depends on the input only through the leading coordinates, $u_k(x) = h_k(U_{1:k}^\top x)$. It still maps into all $d$ dimensions, so $\mathcal{L}_\pi u_k$ is well defined and zero-mean. Moving from stage $k$ to $k'$ widens the network's input from $k$ to $k'$ coordinates. The new input weights start at zero, so the stage-$k$ solution is preserved exactly at the moment of growth.
The next stage opens only when the current one passes a held-out test: its variance reduction factor, $\mathrm{Var}[f]/\mathrm{Var}[f-\mathcal{L}_\pi u_k]$, must have plateaued. This continuous signal plays the role of the verifier assumed in curriculum theory. The stage step $k\to k'$ is chosen so that each stage adds a fixed fraction of the eigenvalue mass of $C$.
The claim is specific and falsifiable. On integrands whose eigenvalue spectrum decays, the curriculum should beat both full-dimensional training and a static top-$k$ projection on error per integrand evaluation, and the advantage should grow with $d$. On integrands with a flat spectrum, it should give no benefit.
Where This Sits
Three works motivate the problem by showing that neural or meta-learned control variates lose ground as dimension grows. Stein Networks stop at $d=20$. Meta-learning Control Variates: Variance Reduction with Limited Data learns an initialization across a uniform distribution of related tasks and reports that it degrades with dimension. Multilevel Control Functional telescopes an integral across hand-defined fidelity levels using kernel control functionals. It inherits cubic kernel cost and notes that its speedup weakens in high dimension.
Our staging is not across fidelity levels or across a task distribution. It runs across the coordinates of a single integrand, and it uses a neural network, so there is no Gram matrix.
The constrained-active-subspace work supplies the ordering of directions. It performs a one-shot rotation and analytic pre-integration followed by quasi-Monte Carlo, which spreads points more evenly than random sampling. We do not pre-integrate analytically. We reuse its eigenvector ordering to schedule what the network sees.
A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula supplies the hypotheses we test, not a proof we inherit. It shows that staged training beats a uniform mixture when three conditions hold: consecutive stages are moderately separated, the sample budget exceeds a threshold, and the initial model can pass the easiest stage. Its setting is maximum-likelihood self-improvement with binary verifiers. We therefore treat its predictions as analogies to check. The first is that the benefit should peak at an intermediate step size. The second is that the curriculum should fail when stage one yields no variance reduction. The paper's own failure case motivates the second check: Llama-3.2-1B regressed by 0.28% on GSM8K.
Self-Paced Learning for Latent Variable Models grows the training set from easy to hard examples. It is the closest classical precedent for staging an optimizer to avoid bad local optima.
Research Program
The first component is the estimator. We build a Stein-layer network with a growable input projection. Each stage is trained on one shared pool of integrand evaluations with Adam, and separately with L-BFGS. The final $\hat I$ is always computed on independent samples.
The second component is the ordering. We estimate $C$ by automatic differentiation where $\nabla f$ is available. Where it is not, we use the gradients of the fitted control variate itself, re-estimating $U$ after each stage. This addresses the cost of computing $C$ that the constrained-active-subspace paper leaves open.
The third component is analysis. We will measure how the stage step size, the eigenvalue decay rate of $C$, and the per-stage evaluation budget jointly determine the final variance. The aim is a simple rule: advance when the held-out variance reduction stops improving by more than a set tolerance. We will also try to characterize, at least empirically, when warm-starting from stage $k$ lands closer to a good stage-$k'$ solution than a fresh initialization does.
Baselines
The naive reference is plain Monte Carlo with the same number of integrand evaluations. All error reduction factors are reported relative to it, matching the metric used in the constrained-active-subspace paper.
The strongest neural competitor is the full-dimensional Bayesian Stein Network trained with L-BFGS, exactly as in Baysian numerical integration with neural networks. That paper reports matching kernel Bayesian quadrature accuracy on 20-dimensional Genz functions while scaling to about $10^6$ points, but it does not quote exact error values. We therefore found no reported numerical result on Lacuna for Stein Networks on Genz and will rerun it ourselves.
Fairness rules apply to every neural method. All of them get the same architecture width, the same total integrand evaluations, the same wall-clock budget including the cost of estimating $C$, and the same hyperparameter search budget, with 20 configurations each. Beating this baseline means a statistically significant improvement in error reduction factor at matched evaluations, over 20 seeds, at $d\ge 50$.
The strongest non-neural competitor on finance and chemistry integrands is the constrained-active-subspace method with quasi-Monte Carlo. It reports error reduction factors of about 7,600 on spread options, about 613 over standard quasi-Monte Carlo on the Heston stochastic-volatility model, and about 2,100 on the chemical Langevin equation (Conditional Quasi-Monte Carlo with Constrained Active Subspaces). We do not expect to beat it on problems where its analytic pre-integration applies. There, success means getting within a small factor of it without a hand-derived tractability template, and matching or exceeding plain active subspaces at about 300 on spread options.
Meta-learned control variates are a further baseline on Lotka-Volterra, where they report 5 to 12 times improvement over the No-U-Turn sampler at small sample sizes (Meta-learning Control Variates: Variance Reduction with Limited Data).
Two ablations isolate the key ingredients. The static ablation trains a single network on the final top-$k$ coordinates from scratch, with no stages, and so tests whether staging matters beyond projection. The order ablation runs the same stage schedule with randomly ordered coordinates instead of active-subspace order, and so tests whether the ordering matters.
Why not just…?
Why not just use L-BFGS on the full network, as Stein Networks already do? This is the most serious threat, because L-BFGS may already handle the narrow landscape. Its memory and per-iteration cost grow with the number of parameters and data points, and the published evidence stops at $d=20$. Our experiments run both methods with both optimizers. If full-dimensional L-BFGS matches the curriculum at $d=100$, the proposal's claim fails, and we will report that.
Why not just project once onto the top-$k$ active directions and fit there? The constrained-active-subspace paper shows that the choice of directions matters a great deal. Unconstrained reduction reached an error reduction factor of about 10 on the chemical network, against about 2,100 for the full method. That evidence concerns which directions are kept, not staging. The static ablation answers whether staging adds anything beyond choosing directions. If it matches the curriculum, the right conclusion is "active subspace plus neural control variate," and we will say so.
Why not just anneal the integrand's smoothness, or train on a uniform mixture? Smoothing changes the target, so a separate correction would be needed to keep the estimate unbiased. Our staging changes only the function class, so the estimate stays unbiased. Uniform mixtures are what the curriculum theory paper shows to be dominated under moderate separation. That evidence comes from reasoning tasks, so we include a uniform baseline: full input from the start, with input dropout across coordinates.
Why not just run adaptive MCMC with more samples? When integrand evaluations are expensive, the control variate literature in this file reports large gains over the No-U-Turn sampler at small budgets. One example is the 5 to 12 times figure above. Another is that the wind-farm integrand used in the Stein Network paper costs 130 seconds per evaluation. Plain Monte Carlo at matched evaluations is our floor regardless.
Proposed Experiments
The first experiment uses Genz's six function families, which test oscillatory, peaked, smooth, continuous and discontinuous behavior. Following the Stein Network and BART-Int papers (Bayesian Probabilistic Numerical Integration with Tree-Based Models, which uses mean absolute percentage error up to $d=10$ and quotes no exact numbers), we run $d\in\{10,20,50,100\}$. We use rotated variants with controlled eigenvalue decay and isotropic variants with a flat spectrum. The decisive check is the interaction between method and spectrum. The curriculum's advantage over full-dimensional training should grow with $d$ under decaying spectra and vanish under flat ones. A uniform gain everywhere would point to an optimization trick rather than the mechanism we claim.
The second experiment uses spread options, the Heston model and the chemical Langevin equation from the constrained-active-subspace paper, comparing against its reported factors.
The third experiment uses Lotka-Volterra parameter posteriors and Sarcos hierarchical-Gaussian-process marginalization from the meta-learning paper. There we also test combining the two ideas: meta-learn only the stage-one network, then grow it.
Two further checks probe the curriculum-theory analogies. A step-size sweep tests whether the benefit peaks at intermediate eigenvalue-mass increments, the moderate-separation prediction. A deliberately poor stage-one ordering tests whether the curriculum collapses when its first stage yields no variance reduction, the feasible-initialization prediction.
References
- Baysian numerical integration with neural networks
- Conditional Quasi-Monte Carlo with Constrained Active Subspaces
- Meta-learning Control Variates: Variance Reduction with Limited Data
- Multilevel Control Functional
- A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula
- Self-Paced Learning for Latent Variable Models
- Bayesian Probabilistic Numerical Integration with Tree-Based Models