Learning the Prior Strength of Northern Carbon Inversions from Held-Out Towers

Estimates of how much carbon northern forests absorb each year come largely from atmospheric inversions. An inversion starts from a model's first guess of surface fluxes, the "prior," and adjusts it until simulated wind-blown CO₂ matches what measurement towers record. The answer depends heavily on a few dozen numbers that practitioners set by hand. These include how far the adjusted fluxes may stray from the prior in each biome, over what distances and time spans those adjustments are correlated, and how much each tower's readings are trusted. The European intercomparison EUROCOM ran six inversion systems on the same 39 stations. It found large spread in country-scale and seasonal fluxes, and much of that spread traced to these subjective error settings. Where towers are sparse, as in boreal Russia and Canada, the same settings decide whether the solver invents "dipoles," paired spurious sources and sinks that cancel in the atmosphere (Bastos et al.; Molina et al.). In short, the northern sink estimate carries a hyperparameter sensitivity that nobody measures and nobody optimizes.

We propose to treat those settings as learnable. They would be chosen to predict CO₂ at towers the inversion never saw. We would also let the prior-error model have heavy tails, because the prior errors themselves are heavy-tailed: Chevallier et al. (2006) compared the ORCHIDEE ecosystem model against 34 Northern Hemisphere flux towers and found Cauchy-like departures with about 30% autocorrelation remaining at 30 days. The machinery is bilevel optimization. An inner problem solves the inversion for fixed settings. An outer problem adjusts the settings by gradient descent on held-out-station error. Step sizes in the outer loop adapt on their own, so the tuning procedure does not introduce new hand-set constants. The deliverable is twofold. First, a number: how much of the inferred boreal and European sink moves when solver settings move within defensible ranges. Second, a procedure that shrinks that range by letting withheld data choose the settings.

Core Claim

A Gaussian prior with hand-set scales has two effects when true prior errors are heavy-tailed. It over-shrinks real large anomalies such as fire, drought, and management pulses. It also spreads the required correction as smooth, wrong structure, which in sparse networks becomes dipoles. We claim two things. The first is that fitting a heavy-tailed prior's scales, tail weight, and correlation lengths per biome, by minimizing error at withheld stations, yields lower flux error and weaker dipoles than expert-set covariances. The second is that it also beats the standard data-driven alternative, which tunes Gaussian covariances by maximum likelihood on the assimilated data.

This differs from the closest work in specific ways. Chevallier et al. diagnosed the heavy tails but explicitly stopped at the observation that non-quadratic inversions are computationally harder. EUROCOM documented covariance-driven spread without proposing how to set the covariances. The bilevel methods we borrow were tested only on text-label cleaning and synthetic problems (Wu et al.).

Research Program

We work in the linear, precomputed-footprint form of regional inversion. Lagrangian transport runs give a matrix $H$ that maps gridded flux corrections $x$ to mole-fraction changes at stations. With $H$ precomputed, one inversion costs matrix products rather than transport runs, which makes both our method and brute-force search affordable. That is deliberate, because it lets the search baselines compete fairly.

For hyperparameters $\theta$, the inner problem is

$$x^*(\theta)=\arg\min_x \tfrac12 (y-Hx)^\top R_\theta^{-1}(y-Hx)+\sum_i \rho_{\nu_{r(i)}}\!\big((C_\theta^{-1/2}x)_i/s_{r(i)}\big)+\tfrac{\epsilon}{2}\|x\|^2.$$

The symbols are as follows. $y$ is the assimilated station data, and $R_\theta$ is the observation-error covariance with a learned scale per station class. $C_\theta$ is a space–time correlation operator with learned lengths, so the heavy-tailed penalty acts on decorrelated increments rather than raw fluxes. $\rho_\nu(u)=\tfrac{\nu+1}{2}\log(1+u^2/\nu)$ is the Student-t penalty with tail parameter $\nu$. Both $\nu$ and the scale $s$ are learned per biome $r$. The small ridge $\epsilon$ keeps the inner Hessian positive definite. In total there are roughly 20–40 hyperparameters.

The outer objective is a robust mismatch, $\sum \rho(y^{\text{held}}-H^{\text{held}}x^*(\theta))$, over stations withheld in leave-station-out folds. We withhold whole stations because neighboring records are correlated, and withholding only time slices would leak information.

The Student-t penalty is nonconvex. We solve the inner problem by iteratively reweighted least squares warm-started from the Gaussian solution. We take hypergradients by implicit differentiation at the found minimum, using conjugate-gradient solves with the inner Hessian, as in Far-HO-style bilevel tooling (Far-HO). A convex Huber penalty serves as a fallback, and a check below tests whether nonconvexity matters.

The outer loop samples random folds and time windows, so its gradients are noisy and its smoothness is unknown. We therefore update $\theta$ with AdaGrad-Norm steps, which Attia et al. prove converge without knowing smoothness or noise level under unbounded gradients. We stop the outer loop with a trajectory-based rule in the spirit of Aolaritei et al. rather than a fixed iteration budget. If hypergradient noise proves heavy-tailed, a normalized-step variant follows Hübler et al.. These optimization papers supply the method; they are not the contribution. We would not use the lower-level theory of UniBiO. It assumes uniform convexity with a known exponent, which our Student-t inner problem violates, so its guarantees constrain what we can claim rather than support it.

Baselines

The naive reference point is the expert-set Gaussian inversion as practiced in EUROCOM: prior scales and correlation lengths taken from the participating systems' published choices. Its reported outcome is a 2006–2015 European NEE sink of −0.15 ± 0.08 Pg C yr⁻¹ (EUROCOM). That is an estimate, not a skill score. No reported result was found on Lacuna for held-out-station mole-fraction error or dipole strength of any EUROCOM system, so we will compute these ourselves on identical footprints and data. The prior-error scale it must correct is about 2 gC m⁻² day⁻¹ (Chevallier et al.).

The strongest existing competitor is a Gaussian inversion whose scale and correlation settings are tuned by maximizing the marginal likelihood of the assimilated data. This is the standard data-assimilation route to objective covariances, and it needs no held-out data. No paper in the file reports it, so we implement it ourselves on the same footprints and parameterization.

The second strong competitor is black-box search, meaning random search and a direct-search tuner (PyHopper). It optimizes the same held-out objective over the same heavy-tailed parameterization and receives exactly as many inner solves as our method consumes, counting each conjugate-gradient hypergradient solve as one inner solve.

Two ablations isolate the ingredients. One uses a Gaussian prior with our held-out bilevel tuning, which isolates the value of heavy tails. The other uses a heavy-tailed prior with expert-set scales, which isolates the value of tuning. Beating a baseline means lower true-flux error in synthetic tests and lower held-out-station error on real data, with non-overlapping fold-bootstrap intervals. A method that wins on held-out error while worsening the dipole index does not count as a win.

Why not just…?

Why not just grid-search the prior scale? For one scalar, a grid is cheap with precomputed footprints, and we expect it to roughly match bilevel tuning. The claim concerns the 20–40 biome-, season-, and station-class-specific settings, where grids become infeasible. There, random search and direct search remain the real threat. The matched-budget comparison settles it: if search at equal inner solves matches us, the contribution reduces to the heavy-tailed, held-out objective, and we will report it that way.

Why not just tune Gaussian covariances by maximum likelihood? This is the most serious alternative. Maximum likelihood rewards covariances that make the assimilated data look typical. Under a Gaussian model it cannot represent the Cauchy-like departures Chevallier et al. measured, so it inflates variances everywhere to absorb a few large errors. Held-out prediction does not share this failure. The synthetic experiment, where the true errors are known to be heavy-tailed, directly tests whether this matters for flux error. The Gaussian-plus-bilevel ablation shows whether any gain comes from the tails or merely from held-out selection.

Why not just use a robust prior solved with L-BFGS and skip bilevel machinery? We do use a standard robust solver for the inner problem. The open question is not how to minimize a Student-t objective but how to set its scales and tail weight. A robust prior with hand-set scales is exactly our second ablation. If it performs as well as the learned version, the bilevel layer is unnecessary.

Proposed Experiments

The first experiment is a synthetic test (an "observing-system simulation") on the EUROCOM station geometry, in the style of the CERES synthetic study (Lauvaux et al.). "True" fluxes are the ORCHIDEE prior plus errors drawn from a Cauchy law with the Chevallier et al. location, scale, and temporal correlation. Pseudo-observations are generated through $H$. We then thin the network from 39 stations to 10 to mimic boreal sparsity. The metrics are flux RMSE against truth at grid and country scale, and a dipole index defined as the negative correlation between corrections in adjacent regions. The decisive check here: at 10 stations, the method should cut the dipole index and country-scale error relative to maximum-likelihood Gaussian tuning. If the method's advantage appears only in the dense network, the core claim fails.

The second experiment uses real EUROCOM data for 2006–2015. We report leave-station-out mole-fraction error and agreement of daily NEE with independent eddy-covariance towers not used in the inversion. We also report the spread of the continental and country sinks across the hyperparameter ensemble our method explores, set against the spread across EUROCOM's six systems. The key quantity is the fraction of inter-system spread attributable to covariance settings alone.

The third experiment transfers to the sparse boreal Eurasian network used in the Russian budget study (Dolman et al.). The check is whether the learned settings move the regional sink outside the hyperparameter-induced range of the hand-set system. As a loose external check, we compare the biome-scale sign of inferred uptake with L-VOD aboveground-carbon trends (Wigneron et al.). This comparison is only indicative, because L-VOD tracks woody stock change rather than net exchange. We would therefore treat agreement as supporting evidence, not a target.

The final check repeats the synthetic experiment with the convex Huber penalty to test whether inner-problem nonconvexity is responsible for any failure. The Student-t runs also start from multiple initial points, so we can detect whether the inner solver lands at different local minima.