Is the Data Plateau Real? Partition-Robust Learning Curves for Where to Collect More Data

When a model underperforms for some population, the usual remedy is to collect more data, and the usual way to decide whether more data still helps is to draw a learning curve: error against training-set size, measured group by group. For geographic data, such as mortgage approvals, crop maps, or school outcomes, "group" usually means a spatial unit like a census tract, county, or grid cell. Geographers have long known that statistics computed over such units change when the units are resized or their borders are redrawn. This is the modifiable areal unit problem (MAUP). This note argues that learning curves inherit the same instability.

A coarse unit that mixes a well-served majority neighbourhood with an under-served minority pocket will look as if extra data has stopped helping, while the pocket's own curve is still falling steeply. A very fine unit will look noisy and miscalibrated whatever we do. If this is right, then "diminishing returns" for a population is partly a property of how we drew the map, and data-acquisition decisions made from one official partition will send samples to the wrong places.

Core Claim

The claim has two parts, and the second is only worth building if the first holds.

The first part is about measurement. The estimated marginal value of additional data for a spatial unit, meaning the slope of its learning curve at the current sample size, is not stable when the zoning or scale of the partition changes. The instability is systematic, not random: aggregating heterogeneous sub-populations makes the curve flatten early, because a unit's error is a sample-weighted mixture of its sub-populations' errors, and the majority's already-saturated term dominates the mixture.

Write $e_u(n)$ for the error of unit $u$ when it holds $n$ training samples. The working model is a per-unit power law $e_u(n) \approx a_u + b_u n^{-c_u}$, where $a_u$ is the irreducible floor, $b_u$ the scale, and $c_u$ the decay rate. This is the concave, saturating shape that Optimal Pricing of Internet of Things: A Machine Learning Approach uses for service quality as a function of data volume. A mixture of such curves with different $c_u$ is not itself a power law, and fitting one to it underestimates the slope of the minority component.

The second part is a method: partition-robust data acquisition. We choose where to collect new samples by minimizing worst-unit error averaged over a distribution $\Pi$ of plausible partitions, rather than over one fixed map. Let $m$ be an allocation of a collection budget $B$ across fine spatial cells, and let $\pi \sim \Pi$ be a random partition drawn by zoning and scale perturbation. Then

$$ m^\star = \arg\min_{m:\ \sum m = B} \ \mathbb{E}_{\pi \sim \Pi} \Big[ \max_{u \in \pi} \hat e_u(n_u + m_u) \Big], $$

where $n_u$ and $m_u$ are the existing and newly allocated samples falling in unit $u$ of partition $\pi$, and $\hat e_u$ is a learning curve fitted at the finest cell level and re-aggregated to each $u$. Because curves are fitted on fine cells and only aggregated afterwards, the method never trusts a coarse unit's apparent plateau.

The closest existing work treats space as a distribution over partitions, but only for reweighting a fixed training set. Fairness by "Where" (SPAD) samples a grid partition each epoch and raises the learning rates of regions that perform poorly. It never asks whether more data would help, and its partitions are regular grids. Our proposal moves the partition distribution from the loss to the data-collection decision, and adds learning-curve estimation as the object that must be made robust.

Research Program

The work starts with a controlled simulation that tests the aggregation-artifact mechanism directly. We build synthetic spatial populations in which the floor, scale, and decay rate of each fine cell's curve are known, and in which a minority population is residentially clustered. We then measure how often single-partition curve fitting misranks units by marginal value, as a function of partition scale and of segregation strength.

Fair Spatial Indexing proves the Partitioning Paradox: calibration error potential grows as units shrink. This constrains the design. Fine cells give unbiased but high-variance curves, so we fit hierarchically, partially pooling each cell's curve parameters toward its spatial neighbours. Spatial autocorrelation justifies borrowing strength from nearby cells.

The second stage measures the instability on real data by subsampling learning curves. We train the same model on nested random subsamples of increasing size, fit curves per unit under many partitions, and quantify partition sensitivity. The sensitivity measure is the rank correlation of marginal-value orderings across partitions, plus the fraction of units whose verdict flips between "saturated" and "still improving." This mirrors the boundary-volatility analysis of Mitigating allocative tradeoffs and harms in an environmental justice data tool. That paper showed that plausible indicator changes flip disadvantaged-community status for 16.1% of California census tracts. We apply the same kind of audit to the data-value estimates themselves rather than to burden scores.

The third stage is the acquisition method and its evaluation in a pool-based simulation. Zoning perturbations are tract agglomerations and Voronoi-style regionalizations. Scale perturbations move between tract, multi-tract, and county levels.

The method also has to respect a result from the curation literature, which constrains how we pick individual samples within a cell. Navigating Towards Fairness with Data Selection found that choosing high-loss "hard" samples tends to mine label bias. We therefore allocate at the cell level and sample uniformly within cells, rather than ranking individual points by loss.

Baselines

The naive reference point is uniform random acquisition from the remaining pool, which is what "just collect more data" means in practice. It is run with the same model, budget, rounds, and hyperparameter search as every other arm. It establishes the floor that any targeted strategy must clear on worst-unit error.

The strongest simple competitor is stratified acquisition by protected group and region: split the budget equally across race-by-county strata, or in inverse proportion to current stratum size. This is the textbook response to minority under-sampling. If the minority pockets coincide with administrative strata, it could match our method, so it is the baseline we most need to beat.

The strongest existing method that uses the partition distribution is SPAD from Fairness by "Where", run on the uniformly acquired data with its Referee-Player reweighting. On Sentinel-2 crop mapping that paper reports a mean absolute distance of regional performance $S(d)_{mean} = 1.094$ against 1.379 for a plain network, with global F1 held at 0.573. We reproduce that setting first. This comparison asks whether reweighting the data we already have closes the gap that targeted collection closes. To keep it fair, SPAD gets the same total sample count and the same tuning budget. Combining SPAD with our acquisition is also reported, since the two are compatible.

The key ablation is single-partition learning-curve acquisition, which is the obvious idea at this intersection. It fits per-tract curves and allocates to maximize marginal error reduction on tracts only, with every other component held identical. Beating it is the direct test of the core claim.

We say our method beats a baseline only if two things hold together. First, it lowers worst-unit error averaged over held-out partition families never used during allocation. Second, it does so without a drop in global accuracy beyond seed noise, measured over at least five seeds per arm.

Why not just…?

Why not just stratify the new data by demographic group? This is a real threat, and the experiments are built to settle it. Stratification fixes under-sampling only along the axes we stratify on. The mechanism here concerns pockets that administrative strata slice through: Auditing for Spatial Fairness found an 84%-approval loan cluster in Northern California, against a 62% average, that fixed borders did not reveal. If stratified acquisition matches ours on held-out partitions, the honest conclusion is that cheap stratification suffices, and we will report that. The synthetic segregation sweep predicts where the two should separate: when minority clusters straddle stratum borders.

Why not just reweight the existing data, as SPAD does, instead of collecting more? Reweighting cannot create information about a pocket that holds a handful of samples. The file's open problems on population bias note exactly this limit for very small sub-cohorts. FairDiffusion, in FairDiffusion, also finds that disparities are not purely a sample-size effect, which cuts the other way. The SPAD-versus-acquisition comparison at matched sample counts measures how much of the gap is a weighting problem and how much is a data problem.

Why not just fit learning curves on census tracts, the official unit? If marginal-value rankings turn out stable across partitions, this simpler method is sufficient and the robust objective is unnecessary. The stage-two measurement is designed to falsify our premise before the method is built. It is the go/no-go check, not a formality.

Why not just use uncertainty-based active learning? Point-level uncertainty and loss selection tends to amplify label bias, a pattern shown on CelebA in Navigating Towards Fairness with Data Selection. Mortgage outcomes plausibly carry historical bias of exactly this kind. We include a predictive-entropy arm anyway, and report its effect on demographic disparity alongside worst-unit error.

Proposed Experiments

The primary benchmark is the 2021 HMDA/Bank of America loan application register used in Legally-Compliant Spatial Fairness Framework and Auditing for Spatial Fairness. It has tract geography, protected applicant attributes, and enough records to simulate a large pool. We start from a deliberately skewed seed set that over-represents high-income, majority tracts, then acquire in rounds. There is no reported predictive-model result on Lacuna for approval prediction on this data; the reported figures are audit statistics (disparate impact 0.96, 493 unfair region pairs). We will therefore report the LC-spatial unfair-pair count after each acquisition strategy as a secondary, legally grounded metric.

Sentinel-2 Central Valley crop mapping supplies the continuous-space setting and the SPAD reference numbers. EdGap (Los Angeles and Houston) supplies a calibration-focused test, using the Expected Neighborhood Calibration Error from Fair Spatial Indexing; no reported numeric ENCE result was found on Lacuna for EdGap.

The decisive checks, in order, are the following. In simulation, does single-partition fitting misrank units more as segregation and aggregation scale increase, as the mixture argument predicts? On HMDA subsampling curves, is the cross-partition rank correlation of marginal value low enough, and the saturated/still-improving flip rate high enough, that allocation decisions would actually differ? In the acquisition simulation, does the partition-robust allocation reduce worst-unit error on held-out county and Voronoi partitions relative to both stratified acquisition and the single-tract ablation, at equal budget? A negative answer to the first two questions is itself a useful result: it would tell practitioners that tract-level learning curves are safe to use for deciding where to collect.