Choosing Spatial Strata by Delphi When Data Are Too Few to Choose

Large environmental datasets are usually split into zones, such as climate regions, soil types or administrative areas, before a model is fit to each zone. The reason is that the same driver can push outcomes in opposite directions in different places, and a single pooled model averages those effects away. The split itself, though, is almost always picked by hand or by a clustering algorithm, and nobody checks whether it was a good choice. At the regional scale this matters most, and it is also where the choice cannot be checked well. With 110 to 248 soil samples per French region, repeated random splits change the measured accuracy of a soil-carbon model by 0.10 to 0.44 in concordance correlation. That result comes from Evaluating validation strategies on the performance of soil property prediction from regional to continental spectral data. With that much noise, picking the zoning scheme that scores best in cross-validation mostly rewards whichever scheme happened to fit the noise in that split.

This note proposes treating the choice of zones as a small-sample model-selection problem, and using the Delphi method to supply a low-variance prior for it. The Delphi method is a structured way of polling experts: they answer anonymously, see the group's summary, and revise over several rounds until their answers stabilise. The claim is testable. When a region has few samples, a zoning scheme agreed by a Delphi panel should give better and more stable held-out accuracy than a scheme chosen by data-driven search over the same candidates. That advantage should shrink and eventually reverse as the sample size grows. If there is no such crossover, the idea is wrong.

Core Claim

The two directions play unequal roles. Climate-based stratification is the modelling decision under study. The Delphi method is the instrument for making that decision when data cannot. The specific hypothesis is a bias-variance crossover in how the zones are selected.

Below a certain number of samples per region, the Delphi-chosen zoning should beat a zoning chosen by nested cross-validation from the same candidate set. Above that sample size, data-driven selection should catch up or win. The comparison is on held-out Lin's concordance correlation coefficient (CCC) for soil organic carbon (SOC). The reason for expecting a crossover is simple. The error from selecting on noisy estimates falls as samples accumulate, while an expert prior's bias stays fixed.

What would be new has two parts. The first is using a Delphi panel to set a structural hyperparameter of a spatial machine-learning model, rather than to write guidelines. The second is testing the panel's output against held-out data under a pre-registered protocol, instead of reporting only the level of agreement.

Relation to Prior Work

Two papers supply the method templates.

Metrics reloaded ran a multi-stage Delphi study with 73 experts and reached 93% consensus on decision rules for choosing image-analysis metrics. It explicitly leaves regression and continuous spatial fields out of scope. It also measures success by consensus, not by downstream accuracy. We borrow its protocol and replace the success criterion with held-out prediction error.

Risk assessment of tunnel water inrush based on Delphi method and machine learning feeds Delphi-derived factor weights into a predictive model to cope with sparse records. It is the closest existing Delphi-plus-machine-learning coupling. However, it weights input features in a single local setting. It does not choose spatial partitions, and it does not compare against a data-driven way of making the same choice.

Three papers motivate the gap.

Two papers constrain the design.

  • TIML conditions a meta-learner on task metadata. It shows that learned, task-aware modulation can beat both simple metadata concatenation and fixed pooling on very small tasks, such as Brazil coffee with 26 positives. This is a warning that a learned soft stratification may make hard expert zones unnecessary. We test that directly.
  • The global carbon sink potential of terrestrial vegetation can be increased substantially by optimal land management shows the common practice of hand-set, uncalibrated stratum parameters: landscape-vegetation-soil zones with a 90th-percentile benchmark within 20 km. It is a cautionary example rather than a testbed, because it has no ground truth to evaluate against.

Research Program

Domain and candidate space. The primary domain is prediction of SOC from visible-near-infrared spectra in the LUCAS topsoil survey. We reuse the setting of the soil validation study: continental Europe, France, and five French regions of 110 to 248 samples each. The model is a Cubist piecewise-linear regression tree.

We first define a finite, shared space of candidate stratifications that both experts and algorithms choose from. Candidates vary three things:

  • the variables that define strata: climate from ERA5 joined by coordinates, elevation, land-cover class, and spectral-derived soil groupings;
  • the number of strata;
  • a minimum number of samples per stratum below which a stratum is merged into its neighbour.

Sharing this space is what makes the comparison fair. Experts and algorithms differ only in how they pick, not in what they can pick.

Delphi panel. We will recruit 25 to 30 soil scientists and pedometricians. We deliberately draw them from at least four countries and from both academic and survey-agency backgrounds, to reduce the risk of a panel whose members share the same blind spots.

  • Round one: each panellist anonymously proposes a stratification from the candidate space for each region and gives a written rationale.
  • Round two: panellists see the median choice, the interquartile range of stratum counts, and anonymised rationales from outliers.
  • Round three: a final revision.

Consensus is declared when at least 75% of panellists endorse the same stratification-defining variables and the interquartile range of stratum count stops shrinking between rounds. This stopping rule is fixed before the panel starts.

Feedback arms. We run two arms with disjoint panellists.

  • The evidence-blind arm receives only group summaries.
  • The evidence-fed arm additionally sees, in round two, repeated-split CCC with intervals for the leading proposals. These are computed on calibration data only.

The contrast tells us whether feeding model output into a Delphi process helps experts, or simply pulls them toward the same noisy estimates that cross-validation already provides. No panellist ever sees the 20% held-out test sets. The regional choices are frozen and published before any held-out evaluation.

Secondary replication. We repeat the design on CropHarvest. There, a smaller panel of agronomists groups source tasks into agroecological strata for pretraining. The three target tasks are Togo crop versus non-crop, Kenya maize, and Brazil coffee. The goal is to see whether the crossover pattern appears outside soil science.

Baselines

Naive reference. A single Cubist model fit to each region's calibration data with no stratification, and a single France-wide unstratified model. These tell us whether stratifying helps at all. At national scale the file reports an average SOC CCC of about 0.83 in the soil validation study. We treat this as the reference level that a regional method should approach. At regional scale the same paper reports only the spread of CCC across 50 repeated splits (0.10 to 0.44), not a single mean. Beating the naive reference means a paired mean CCC gain whose 95% interval over 50 repeated splits excludes zero.

Administrative-region stratification. The five French regions are used as given, which is how the soil validation study and the Chinese yield study partition space. This is the default practice. It receives the same Cubist tuning budget as every other method.

Automated clustering. k-means and Gaussian mixtures on the same covariates the experts may use, with the number of clusters chosen by an information criterion. This is the direct form of the "just cluster it" objection. The clustering literature above suggests it is sensitive to parameter choices, so we report results over the full range of reasonable settings rather than one lucky configuration.

Nested-cross-validation search over the candidate space. This is the strongest competitor. It sees exactly the options the experts see and picks the one with the best inner-loop CCC. The outer loop uses 50 repetitions, matching the soil validation study, so selection noise is measured rather than hidden. It receives the same compute budget and the same Cubist tuning grid as the Delphi-chosen configuration. To beat it, the Delphi configuration must show both of the following in at least three of five regions:

  • higher mean held-out CCC;
  • lower variance across repeated splits.

Learned soft stratification. An unstratified neural or Cubist model given coordinates and climate as inputs. On CropHarvest, this baseline is TIML itself, with its forgetfulness rule. The file reports that TIML achieved the highest average AUC ROC and F1 across the three tasks. It also reports that TIML was the only transfer model to beat random initialisation in F1 on Brazil coffee. No reported numeric AUC or F1 value for TIML was found on Lacuna, so we will rerun TIML from its published configuration and treat that rerun as the target.

Ablations isolating the Delphi ingredient. Two ablations separate the value of experts from the value of the Delphi process.

  • A single-round expert average: round-one answers aggregated by majority, with no iteration or feedback. This separates expert knowledge from Delphi's iterative refinement.
  • The evidence-fed versus evidence-blind contrast described above.

Why not just…?

Why not just cluster the covariates automatically? Clustering is cheap and reproducible, and experts are neither. The clustering benchmark cited above found that results vary more with parameter settings than across methods, and that expert assessments diverge from standard cluster-quality scores. The more important point is that a clustering algorithm optimises how compact the clusters are in covariate space, not how well the resulting models predict SOC. We do not argue this point in the abstract. The clustering baseline is run on the same candidate covariates, and if it matches Delphi at regional scale, the expert step is not worth its cost.

Why not just pick the stratification that cross-validates best? This is the real threat, and the whole design is built around it. Our argument is that with 110 to 248 samples, split-to-split variation of 0.10 to 0.44 in CCC means the winner of the inner loop is partly chosen by noise. Nested selection should therefore look optimistic in the inner loop and underperform in the outer loop. If nested search matches or beats Delphi even in the smallest regions, the core claim is false and we will report that. The subsampling experiment below also shows where the two approaches cross.

Why not just give a flexible model coordinates and climate, or use TIML? The seven-region Chinese yield study shows sign reversals that a pooled linear model cannot capture. However, a flexible model with soft conditioning, like TIML, may learn such reversals without any hard zones. We include both as baselines. If the soft-conditioning approach wins at every sample size, the conclusion is that hard strata are the wrong object to elicit. In that case the Delphi effort should instead go toward priors on the conditioning variables.

Why not just convene a workshop? An open meeting is faster, but it lacks anonymity and a pre-set stopping rule. Without those, it produces a choice that is hard to audit or reproduce. We do not have the budget to run a workshop arm on the full panel. We therefore treat this as a limitation. The single-round ablation gives a partial answer, because it measures what iteration adds over independent expert opinions.

Decisive Experiments

Subsampling curve. The central test draws calibration sets of 100, 200, 400, 800 and 1,600 samples from the France national data. At each size we compare the Delphi-chosen stratification against nested-cross-validation search over 50 repetitions. The claim predicts a positive Delphi advantage at 100 to 200 samples that falls toward zero or below by the national size. A flat curve would refute the claim, whether it sits above or below zero:

  • flat and above zero means experts are simply better at every scale, which is a different finding;
  • flat at or below zero means experts add nothing.

Stability under resampling. We apply nested search to bootstrap resamples of each region and count how often it picks the same stratification. Delphi's choice is fixed by construction. We will then check whether regions where nested search's choice is unstable are the regions where Delphi gains most. That link is the proposed mechanism, and it can be tested directly.

Transfer across regions. Each region's chosen stratification and model are applied to the other four regions. This checks whether expert zoning captures structure that carries across places or only local fit.

Replication on CropHarvest. The same comparison against TIML on the three target tasks checks whether the pattern appears beyond soil spectroscopy.

All analysis code, frozen panel outputs and stopping-rule logs will be released before held-out scores are computed, so the evaluation cannot be tuned after the fact.