Letting the Carbon Budget Grade Climate Clusters

Each year the rate at which carbon dioxide accumulates in the atmosphere swings by billions of tons, even though human emissions rise smoothly. Most of that swing comes from land vegetation, and specifically from dry grasslands and savannas that absorb carbon in wet years and release it in dry ones. To study these swings, Earth scientists divide the planet into regions: fixed aridity bands, satellite land-cover classes, or continents. Machine learning offers "parameter-free" clustering methods, meaning algorithms that choose the number and shape of groups without anyone tuning knobs. These methods could draw the regions from the data instead. The difficulty is validation. Clusters of climate data have no ground-truth labels, and parameter-free algorithms are almost never tested on real geophysical fields, so nobody knows whether data-drawn regions are real or artifacts.

This note proposes a way out. The measured global growth rate of atmospheric CO₂ serves as an independent scorekeeper. A regionalization is better if, after calibrating each region's modeled carbon response on some years, it predicts the atmospheric growth rate in years it has never seen more accurately than a fixed aridity or biome map with the same number of regions. We pair this real-data test with a synthetic benchmark that plants known regions into real historical climate fields. We then run one specific parameter-free method through both tests: a signed similarity graph between grid cells, cut by the Mutex Watershed. The goal is to find out whether data-drawn regions earn their keep, and to see where the tropical water–carbon coupling shift that current models miss is actually located.

Core Claim

The claim has three parts. The first is a claim about the problem. For interannual carbon variability, the value of a regionalization can be measured without labels. The test is how well regional calibration of land-model fluxes predicts held-out atmospheric CO₂ growth-rate anomalies. The global sum of fluxes does not depend on how the land is partitioned, so any gain must come from grouping together cells whose model errors are shared. That is exactly what a physically meaningful region should do.

The second part concerns the method. We build a graph over land grid cells. Two cells are joined by an "attractive" edge when their year-to-year flux or water responses rise and fall together, and by a "repulsive" edge when they move in opposite directions. The boundary between attractive and repulsive is set by a surrogate-data null rather than by a hand-set cutoff. The graph is then partitioned with the Mutex Watershed, a greedy algorithm that sorts edges by strength and merges or separates cells without seeds, thresholds, or a preset number of clusters. We predict that this beats static masks on held-out-year growth-rate prediction and on stability across models.

The third part is a localization claim. Wang et al. found that the sensitivity of the atmospheric CO₂ growth rate to tropical water storage became about 35% more negative between 1960–1989 and 1989–2018. They also found that climate models do not reproduce this shift. We predict that the shift concentrates in a small number of data-drawn semi-arid transition regions rather than being spread across the tropics.

Why this gap is real

The carbon-cycle papers in the corpus find strong regional structure but impose it by hand. Zhang et al. weight 14 flux models by agreement with satellite chlorophyll fluorescence and attribute 57% of detrended global photosynthesis variability to semi-arid regions. Their regions are fixed aridity bins and land-cover classes, and they acknowledge a mismatch in spatial scale between models and observations. Fan et al. use L-band microwave biomass data and find that semi-arid biomes drive over 55% of tropical carbon-stock variability and that forest degradation losses are about twice those from deforestation. Their analysis still runs on static continental and biome masks, which cannot see degradation. Bastos et al. compare 16 TRENDY land models against the CAMS and CarboScope atmospheric inversions for the 2015/16 El Niño. The land models give a 1.0 PgC/yr sink reduction and the inversions give 0.4–0.7 PgC/yr, aggregated over hand-drawn tropical zones. Weber et al. apply principal component analysis to African fluxes and find that moisture drives photosynthesis variability. However, the method is linear, the number of components is chosen by hand, and African inversions are inconclusive. Wang et al. collapse the whole tropics into one time series. All of these papers motivate the proposal, because each shows that regional heterogeneity matters and then averages it away.

On the clustering side, parameter-free methods are validated on images, digits, shopping baskets, and social networks. Examples are the Mutex Watershed on electron-microscopy images, the connectivity kernel on USPS digits, txmeans on retail data, and exact-ICL signed block models on political networks. The Nibble local clustering algorithm has no experiments at all, and its authors note that the proven parameter settings are too conservative to use. CSTS builds a "structure-first" benchmark with planted correlation patterns and shows that popular methods break under non-normal data. It also shows that correlation estimates need roughly 30 observations to be reliable. CSTS constrains the design: annual carbon records of about 40 years sit near that limit, and its test series have only three variables. These methods supply the machinery and CSTS supplies the evaluation philosophy. None of them has been tested on the kind of short, spatially smooth, teleconnected fields that carbon science produces.

Research Program

Data and representation. We use 0.5° gridded net biome productivity (net carbon uptake by land) and photosynthesis from the TRENDY-v6 ensemble used by Bastos et al., together with CRU-NCEP climate forcing. For water, we use the GPCC precipitation and water-storage records paired with the NOAA growth rate by Wang et al.. Following Contribution of land use to the interannual variability of the land carbon cycle, which shows managed land drives 30–45% of land-balance variability, we include simulations with and without land use as a separate view. Each cell becomes a vector of detrended annual anomalies.

The claim "parameter-free" usually hides choices in how the data are represented. We therefore fix the detrending method, the anomaly variable, and the correlation measure before any results are seen. We use rank correlation, because CSTS found it robust to non-normal data. We then report how much each of these choices moves the results. This audit of hidden choices is itself a contribution.

Signed graph and partition. Neighboring cells receive attractive edges weighted by the rank correlation of their anomalies. A sparse set of long-range pairs receives repulsive edges when their correlation is significantly different from zero after accounting for effective sample size. Significance is judged against a null built from phase-randomized surrogate series, which removes the need for a hand-set cutoff. The Mutex Watershed cost scales as $O(|E|\log|E|)$ in the number of edges $|E|$, which is tractable for tens of thousands of land cells.

As alternative partitioners on the same graph we run the exact-ICL signed block model, which chooses the number of groups through a Bayesian likelihood. On a 2° coarsened grid we also run the connectivity kernel. At 0.5° its 2-approximation becomes too expensive. Its known weakness on smooth gradients is also the typical structure of climate fields, so this is a direct test of that failure mode.

Synthetic benchmark. Real climate forcing fields are retained, and flux fields are generated from planted regions. Each planted region has its own response rule, for example a Budyko-style water–carbon slope in the spirit of Zhang et al., or a lagged response to ENSO (the El Niño–Southern Oscillation climate cycle). The generator then degrades the data along the axes CSTS identified: record length from 60 down to 15 years, heavy-tailed noise, missing-data masks, and smooth gradient bridges between regions. This extends the structure-first philosophy of CSTS from three-variable series to gridded fields.

Budget-closure test. For a regionalization with $K$ regions, we fit

$$\Delta G_t = a + \sum_{k=1}^{K} b_k\, F_{k,t} + \varepsilon_t$$

on training years. Here $\Delta G_t$ is the detrended NOAA atmospheric growth-rate anomaly in year $t$. $F_{k,t}$ is the ensemble-mean modeled land-flux anomaly (or observed water-storage anomaly) summed over region $k$. The $b_k$ are regional calibration factors. We then score out-of-sample error on held-out blocks of consecutive years. Leaving out whole blocks prevents ENSO persistence from leaking between training and test sets. All methods share one ridge penalty, chosen once by nested cross-validation on the static-mask baseline.

Baselines

The naive reference is no regionalization: a single tropical, or global, sum with one calibration factor. This is effectively the monolithic aggregation used by Wang et al. Any method that fails to beat it has shown nothing.

The strongest domain baseline is static aridity-index bins crossed with land-cover classes, which is the regionalization of Zhang et al.. It is the strongest competitor because it encodes the known physical finding that semi-arid regions dominate. The only reported number is the 57% attribution of detrended photosynthesis variability to semi-arid regions, and it will be reproduced as a sanity check. The second domain baseline is the continental and biome aggregation of Bastos et al. and Fan et al.

The strongest data-driven baseline is k-means on the same rank-correlation features, with the number of clusters chosen by silhouette score. The next is spectral clustering with a Gaussian kernel whose bandwidth is chosen by the same silhouette score. The third is principal component analysis followed by k-means, as in Weber et al.

For fairness, every baseline receives the same features, the same detrending, and the same compute. Each baseline is also run with an "oracle" number of clusters set equal to whatever the Mutex Watershed selects. This isolates whether any advantage comes from choosing the number of regions or from their shape.

No reported result was found on Lacuna for clustering interannual carbon-flux fields or for held-out growth-rate prediction by regionalization. The closest clustering figures, such as txmeans reaching a median NMI of 0.97 on retail data, come from unrelated domains and are not comparable. Beating a baseline means two things: lower held-out growth-rate error, with a paired block-bootstrap confidence interval that excludes zero, at equal $K$; and higher cross-model stability.

Four ablations target the ingredients. The first removes the repulsive edges, which reduces the method to a plain watershed with a threshold. The second replaces the surrogate-calibrated sign with a zero-correlation cutoff. The third swaps rank correlation for Pearson correlation. The fourth clusters a single TRENDY model instead of the ensemble.

Why not just…?

Why not just use aridity bins and land-cover maps? These are physically grounded and already explain 57% of photosynthesis variability in Zhang et al.. They are the main threat. There is evidence they miss something. Fan et al. show that degradation losses, which are about twice deforestation losses, are invisible to static cover classes. The land-use paper shows that static bookkeeping put the land-use share of variability at 5%, against 30–45% when management is resolved. Still, "misses something" does not mean "predicts worse." The budget-closure test at equal $K$ settles the question directly. If static bins match data-drawn regions there, we will report that static bins are sufficient.

Why not just run principal component or EOF analysis? Empirical orthogonal functions (EOFs) are the standard tool for extracting climate modes and are cheap. They produce overlapping, sign-ambiguous spatial patterns rather than regions, and they assume linear, orthogonal structure. Weber et al. needed a heuristic choice of how many components to keep. We include PCA followed by k-means as a baseline. The synthetic benchmark, with its planted non-linear Budyko responses, will show whether the linear assumption actually costs accuracy at realistic record lengths.

Why not just use k-means with a silhouette score or elbow rule? This is simple and may be enough. The evidence against it comes from other domains. Heuristic model selection badly overestimated the number of clusters in the txmeans experiments, and k-means failed on elongated structure with 30.6% error on USPS digits in the connectivity kernel paper. Neither result transfers automatically to climate fields. The oracle-$K$ comparison is decisive here. If k-means given the Mutex Watershed's $K$ matches it, then the only benefit of the proposed method is choosing $K$, and we will say so.

Why not skip clustering and regress the growth rate on every grid cell with a sparse or ridge penalty? For pure prediction this could match or beat any regionalization, so it is a genuine threat. With about 40 years of data and thousands of cells, however, the coefficient maps are likely to be unstable. We run this regression as an upper-reference predictor and measure the stability of its coefficient maps across bootstrap resamples of years. If it predicts as well and is also stable, the case for clustering reduces to interpretability.

Decisive Experiments

The first experiment establishes whether the method recovers known structure. On the synthetic benchmark, we measure normalized mutual information against the planted regions as record length shrinks and gradient bridges are added. The prediction is that the Mutex Watershed with surrogate-calibrated signs keeps high recovery down to about 30 years, consistent with the CSTS reliability threshold. We also predict that the connectivity kernel degrades first on smooth gradients, as its own authors anticipate. If recovery collapses at 40 years, the real-data results cannot be trusted and the proposal becomes a negative result about applying these methods to climate records, which is still worth publishing.

The second experiment is the budget-closure test on 1960–2018, using held-out year blocks. The comparison is against every baseline at equal $K$. Stability is measured by adjusted Rand index between regionalizations built from disjoint halves of the TRENDY ensemble.

The third experiment localizes the coupling shift. Within the best regionalization, we decompose Wang et al.'s change in water–growth-rate sensitivity region by region and apply the same decomposition to the CMIP6 outputs they used. The prediction is that a minority of regions carries most of the observed shift and that models diverge specifically there.

The fourth experiment is a targeted El Niño check. For 2015/16, we ask whether the gap between the land models' 1.0 PgC/yr sink reduction and the inversions' 0.4–0.7 PgC/yr concentrates in data-drawn semi-arid clusters. Where the 2010–2017 L-band biomass anomalies of Fan et al. are available, they provide an independent spatial cross-check.

A clear failure in the second experiment, meaning no gain over aridity bins at equal $K$, would count against the core claim no matter how attractive the resulting maps look.