Testing Whether Satellite Air-Quality Maps Get the Peaks Right

Maps of ground-level air pollution are built by combining satellite measurements, weather data and a sparse set of ground monitoring stations. These maps are compared and chosen almost entirely by one summary number, an $R^2$ or a root-mean-square error averaged over all stations and hours. The people who use the maps care most about pollution episodes, and the literature keeps reporting that models with good averages miss those episodes. A deep network for Wuhan PM2.5 (fine particulate matter) underestimates whenever ground readings exceed about 55 μg/m³. A wildfire-season model for British Columbia estimates an August peak monthly average of 83 μg/m³ while stations record more than 100. A model transferred from Los Angeles to Accra cannot predict concentrations above what it saw in training. We propose a statistical test that asks one question: in a given region and season, does the upper tail of a model's predictions at held-out stations match the upper tail of what those stations measured? Because the test is calibrated for the fact that nearby stations and consecutive hours are correlated, its p-values can be trusted. It can also compare two models' peak fidelity, which an average score cannot do.

The test borrows from nonparametric group comparison, the family of statistics that compare distributions without assuming normality, such as rank tests and Wasserstein distances. Satellite-to-ground fusion is where it is applied. The ingredient we expect to matter most is the null calibration. Classical rank tests assume independent observations, and spatial and temporal autocorrelation makes them reject far too often. A peak-fidelity test that rejects too often is worse than having no test at all.

Core Claim

The test statistic is a tail-restricted quantile distance. Fix an upper quantile level $\tau$, say 0.9. For a held-out station $s$, let $F_s$ be the empirical distribution of its measured concentrations over the evaluation period, and let $G_s$ be the distribution of the model's predictions at the same place and times. The statistic is

$$T_\tau = \frac{1}{|S|}\sum_{s \in S} \int_\tau^1 \big(F_s^{-1}(u) - G_s^{-1}(u)\big)\,du,$$

where $S$ is the set of held-out stations in a region and $F_s^{-1}$, $G_s^{-1}$ are quantile functions. $T_\tau$ is the signed area between the two quantile curves above level $\tau$. It is the upper-tail piece of the univariate Wasserstein distance, which On Wasserstein Two-Sample Testing and Related Families of Nonparametric Tests connects to quantile-quantile plots and distribution-free tests. We keep the sign because systematic underestimation of peaks is the failure we care about. We keep physical units because rank transforms discard the magnitudes regulators need, a limitation that paper's line of work also notes. A paired version, $T_\tau^{A} - T_\tau^{B}$, compares two models on the same stations.

The null distribution is generated by resampling in a way that respects the dependence structure. For the one-model test, the null is "predictions and measurements are exchangeable at each station-time." Within spatial blocks, we swap predicted and observed values over contiguous time blocks, so local autocorrelation is preserved. For the two-model test, we flip the sign of whole spatial-block-by-time-block residual differences. The claim we test is threefold. First, this calibration holds the Type I error rate near nominal on realistically autocorrelated data where independent permutation fails. Second, the resulting peak-fidelity rankings of fusion models differ from their RMSE rankings in ways that matter. Third, robust training losses that look good on average can be shown, with valid inference, to worsen peaks.

Relation to Prior Work

Several papers motivate this proposal by documenting the tail failure under average-score evaluation. These are Deep Learning for Ground-Level PM2.5 Prediction from Satellite Remote Sensing Data, which reports the underestimation above about 55 μg/m³ and attributes it to the mismatch between point stations and 0.05° pixels, Estimating PM2.5 Concentrations in British Columbia during Wildfire Season, and Deep Transfer Learning on Satellite Imagery Improves Air Quality Estimates in Developing Nations. Each describes the peak problem but evaluates with $R^2$ or RMSE and draws no formal inference about it. Fusion papers such as Estimating PM2.5 in the Beijing-Tianjin-Hebei Region Using MODIS AOD Products, Estimation of Ground-Level NO2 in China Using GEMS Measurements and a Nested Machine Learning Model and Fine-Scale and Population-Weighted PM2.5 Modeling in Melbourne supply the models and data we evaluate, and none of them tests differences between models. Benchmarking Regression Models Under Spatial Heterogeneity compares spatial regression with random forests under controlled non-stationarity, using test RMSE only. We reuse its synthetic data generators and add tail structure and a testing layer.

The closest bridge between the two directions is Aspects of weather parameters at Neumayer station, Antarctica. It uses a Mann-Whitney test on skewed wind data to compare a station with reanalysis grid points. That paper does not correct for the 3-hourly autocorrelation, compares medians rather than tails, and covers a single point. We generalize the comparison to many held-out stations, the upper tail and dependence-aware calibration.

Two papers constrain the design. Evaluating Predictive Models of Student Success shows that tests assuming independence give misleading model groupings on correlated cross-validation folds. Its frequentist Nemenyi test lumped 19 models as "best," while a Bayesian correlated test narrowed this to 4. This directly motivates our dependence-aware null. Universal Robust Regression via Maximum Mean Discrepancy shows that bounded kernels make distribution matching insensitive to extreme points. Maximum Mean Discrepancy (MMD) compares two distributions through averages of a similarity function called a kernel. That insensitivity is useful for robustness, but it implies an off-the-shelf bounded-kernel MMD test will be blind to exactly the peaks we target. We therefore use it as a baseline, not as our statistic. The same paper's MMD loss is also one of the training objectives whose tail cost we measure.

Research Program

The work has three parts.

The first is calibration. We use the synthetic generators from the spatial heterogeneity benchmark, with sample sizes from 100 to 5000, and add controlled spatial autocorrelation, temporal autocorrelation and heavy-tailed spike processes. Under the null, predictions and observations share a tail. Under the alternatives, peaks are shrunk by a known factor. We measure Type I error and power for our block-resampled statistic and for each baseline. We also study how block size should scale with the estimated correlation range. A design that is too small inflates error, and one that is too large leaves too few blocks for power. We will report where this trade-off breaks, especially on small networks such as Melbourne's 18 sensors.

The second part is a re-audit of real fusion problems. On each dataset we train a common panel of models under spatial-block cross-validation:

  • global least squares;
  • geographically weighted regression, meaning local regressions centered at each location;
  • spatial random forest;
  • XGBoost;
  • an MMD-loss regressor;
  • a quantile-loss regressor.

On the GEMS data we add the nested XGBoost with predicted mixing height. We then compare RMSE rankings with peak-fidelity rankings, region by region and season by season.

The third part addresses the tension between robustness and peaks, which no prior paper resolves. Bounded-kernel MMD training resists contaminated readings, but it may also suppress genuine spikes. We test whether it does so significantly relative to squared loss, using the paired statistic, while holding RMSE roughly matched.

Baselines

The do-nothing reference is ranking models by held-out RMSE and $R^2$, as every fusion paper in the file does. For context, the strongest fusion results we will try to reproduce are these:

  • nested XGBoost on the China GEMS NO2 data, 10-fold cross-validated $R^2$ = 0.93 and RMSE = 4.19 μg/m³ (GEMS nested model);
  • Geoi-DBN on Wuhan, out-of-sample $R^2$ = 0.80 and RMSE = 17.49 μg/m³ (Wuhan deep learning);
  • geographically weighted regression in British Columbia, 76% explained variability (British Columbia wildfire).

These are the models being audited. The baselines are competing ways of deciding whether a model gets peaks right and which model gets them more right. Our test beats RMSE ranking if it reverses or separates model orderings in ways that later peak periods, not used for the test, confirm.

The naive nonparametric baseline is the analysis the Neumayer paper and much applied work would run: a Wilcoxon or Mann-Whitney test, or a Friedman rank test across models, on per-station errors treated as independent. It gets the same data and the same tail restriction, with only the null calibration changed. This baseline doubles as the ablation that isolates our key ingredient, block-structured resampling. If it holds nominal Type I error on the synthetic suites, our contribution shrinks to the choice of statistic.

The strongest competitor is threshold-conditioned RMSE: error computed only on station-hours whose observed value exceeds a threshold, with confidence intervals from a spatial block bootstrap. It is simple, interpretable and already dependence-aware. We give it the identical blocking scheme and evaluate it at matched thresholds. One structural difference is that conditioning on the observed value exceeding a threshold selects on the outcome. That inflates apparent underestimation through regression to the mean even for a well-calibrated model. Our quantile comparison does not condition on the outcome. We will measure the size of this bias on the synthetic nulls. We will count our test as superior only if it has better-calibrated error rates or higher power at equal error rates.

Two further baselines complete the set. The first is a bounded-kernel MMD two-sample test between predicted and observed values, with block-permutation nulls for fairness. It represents general-purpose distribution tests, and we expect it to have low power on tails for the reason given above. The second is the Bayesian hierarchical correlated t-test from the student-success paper, applied to per-fold tail errors. It represents the existing answer to correlated-fold comparison, though it is parametric.

No reported result was found on Lacuna for the Type I error or power of any of these tests on satellite-ground fusion data. Those numbers will be produced here.

Why not just…?

Why not just report RMSE above a pollution threshold? This is the real threat, and the experiments are built to settle it. It answers a similar question but selects on the outcome, which biases it toward finding underestimation in every model. On its own it is also a point estimate without calibrated uncertainty. If block-bootstrapped threshold RMSE matches our test's Type I error and power on the synthetic suites and yields the same model decisions on real data, we will say so. In that case the deliverable becomes the validated blocking scheme rather than the statistic.

Why not just use a t-test or ANOVA on errors? The student-success paper shows that tests assuming independence collapse distinctions on correlated folds. Pollution errors are additionally skewed and heavy-tailed, which is exactly the regime where the Neumayer authors abandoned means. The parametric correlated t-test is included as a baseline, so this is checked directly rather than assumed.

Why not just use a domain classifier or MMD test to flag shift before deployment, as DeepAQ's adversarial critic implicitly does? Input shift and tail failure are different events. A classifier can detect that Accra imagery differs from Los Angeles imagery without saying whether predicted peaks will be capped. A bounded-kernel MMD test underweights extremes by design. On the Accra transfer data, we will check whether input-shift tests computed per region predict where our tail statistic later rejects. If they predict it well, input-shift testing is the cheaper tool for that use.

Proposed Experiments

The datasets are the Wuhan network (104 stations, hourly Himawari-8 aerosol optical depth, GEOS-5 meteorology), the China GEMS NO2 network (856 stations, which gives power for fine regional splits), British Columbia wildfire season (66 stations), Beijing-Tianjin-Hebei (13 cities) and Melbourne (18 sensors, the small-sample stress case). Synthetic suites come from the spatial heterogeneity benchmark. The Accra transfer data is used for the shift question above.

The metrics are empirical Type I error at the 5% level, power curves against known peak shrinkage, and agreement between test-based model rankings and peak errors on a temporally later held-out episode. We will also count region-season cells where the RMSE ranking and the peak-fidelity ranking disagree significantly.

Three results would be decisive.

  1. On synthetic nulls with realistic correlation ranges, the independence-assuming rank test should inflate Type I error well above 5% while ours stays near nominal.
  2. On Wuhan and British Columbia, our test should reject peak fidelity for models with competitive RMSE, matching the qualitative failures those papers describe. The rejection should be localized by region and season.
  3. The paired test should show whether MMD-loss training significantly worsens upper-tail fidelity relative to squared loss at matched RMSE. Either answer resolves an open tension in the file.

If the calibrated test and block-bootstrapped threshold RMSE agree everywhere, we will report the simpler tool as sufficient.