Most estimates of how much COVID-19 lockdowns cleaned the air, and how quickly pollution rebounded afterward, come from one recipe. A model, often a random forest, is trained on pre-lockdown data to predict pollution from weather and calendar features. It then predicts what pollution "would have been" during lockdown, and the gap between observed and predicted values is reported as the lockdown effect. These effects are usually reported as point numbers, such as a 3.61 µg/m³ drop in PM2.5 over the Yangtze River Delta or a −52.8% NO₂ change in Wuhan, with no interval attached. Random forests offer a cheap way to add one. Each tree is fit on a bootstrap resample, so roughly a third of the data is "out-of-bag" for that tree. The forest's errors on those left-out points can therefore stand in for test errors without holding data aside. The trouble is that pollution series are strongly autocorrelated. A left-out hour is surrounded by in-bag hours from the same weather episode. Out-of-bag error then measures how well the forest fills gaps between known days, not how well it extrapolates weeks into a period it never saw. Intervals built this way should be too narrow. Lockdown effects and recovery dates would then look more certain than they are.
We propose to fix and test this directly. The forest will be resampled in contiguous blocks matched to the length of the effect window, so that its out-of-bag residuals mimic the deployment gap. The resulting effect intervals will be checked on "placebo" windows in pre-pandemic years, where the true lockdown effect is known to be zero. Placebo checks need a calibration diagnostic, and we will use one that does not depend on arbitrary binning. Binned reliability checks are known to hide local failures through cancellation and to report spurious miscalibration through finite-sample noise. The deliverable is a defensible uncertainty statement for lockdown effects and for the date each city's air quality "recovered" to its counterfactual.
Core Claim
The claim has three parts, each testable. First, naive i.i.d. out-of-bag intervals around counterfactual air-quality effects under-cover badly on placebo windows. The coverage of zero should fall far below nominal, especially for effects averaged over weeks, where autocorrelated errors do not average out. Second, block out-of-bag residuals, with block length equal to the effect window, restore near-nominal placebo coverage. They do this without discarding the most recent pre-lockdown data, which matters because Chinese and European pollution was trending down before 2020. Third, whether an interval method looks "conditionally calibrated" across weather regimes and pollution levels depends on how the diagnostic bins the data. A binning-free test on coverage indicators gives a stable verdict where binned checks disagree.
The file contains a motivating number. On the AirDelhi dataset, the best random forest reaches RMSE 22.24 at spatial interpolation, while the best forecasting model (XGBoost) only reaches RMSE 84.15 (AirDelhi). Interpolation error is a poor proxy for extrapolation error, and naive out-of-bag error is an interpolation measurement. The Yangtze River Delta study shows the same pattern: fit R² of 0.938 against cross-validated R² of 0.77 (Lu et al.).
Research Program
The estimand is a window-averaged effect. Let $W$ be a contiguous window of $|W|$ days, $y_t$ the observed concentration on day $t$, $x_t$ the weather and calendar features, and $\hat f$ a forest trained on business-as-usual data. The effect estimate is
$$\hat\Delta_W = \frac{1}{|W|}\sum_{t\in W}\big(y_t - \hat f(x_t)\big).$$
In block out-of-bag, each tree is trained on a resample of contiguous blocks of length $L$ rather than individual days. For every block $b$ that a tree never saw, we record the block-mean residual $\bar r_b$, averaged over the trees for which $b$ was out-of-bag. The $(1-\alpha)$ interval for $\Delta_W$ is $\hat\Delta_W$ minus the $\alpha/2$ and $1-\alpha/2$ empirical quantiles of $\{\bar r_b\}$, with $L$ set to $|W|$ plus a buffer equal to the autocorrelation length.
This construction calibrates the aggregate directly rather than combining per-day intervals. Combining daily intervals under an independence assumption would shrink width like $1/\sqrt{|W|}$, and autocorrelation makes that shrinkage wrong. The method sits on top of the stability-based coverage result of Wang, Wu and Nettleton. That paper proves that random-forest out-of-bag intervals approach nominal coverage under exchangeable data with light-tailed responses. We supply the method in a regime that result does not cover. We test empirically where its guarantee breaks and whether blocking recovers it, rather than claim a new theorem.
The recovery analysis follows from the same machinery. For each city and pollutant, rolling effect intervals are computed through 2020 and into 2021. The recovery date is the first time the interval covers zero for a sustained run. We will report how much that date moves between naive and block intervals. This is the quantity policy discussions of "rebound" actually use, for example the CO₂ rebound described by Jackson et al..
The calibration audit borrows from the binning literature. Kumar, Liang and Ma prove that binned calibration error is a lower bound that hides opposing errors within a bin. Widmann, Lindsten and Zachariah show that plug-in binned estimates flag well-calibrated models because of noise, and they offer unbiased kernel statistics with tractable null distributions. Gupta and Ramdas propose validity plots as a replacement for binned summaries.
All three were developed for classifier probabilities, but the object we need to audit is also binary. It is the coverage indicator $Z_i = 1\{\Delta=0 \in C_i\}$ for each placebo window $i$, which should equal 1 with probability $1-\alpha$ whatever the window's weather regime or pollution level $s_i$. We test this with the U-statistic $\hat T = \frac{1}{n(n-1)}\sum_{i\ne j}(Z_i-(1-\alpha))(Z_j-(1-\alpha))\,k(s_i,s_j)$, where $k$ is a kernel over standardized covariates. This statistic is the same in structure as the kernel calibration error of Widmann et al. It is unbiased and needs no bins. Because placebo windows overlap and share weather, its null distribution will come from a block bootstrap over years rather than from the i.i.d. asymptotics in that paper.
Baselines
The naive reference point is the current practice of point effects with no interval. We will also include the textbook fallback of daily residual standard deviation, scaled by $1/\sqrt{|W|}$ under an independence assumption. It quantifies how wrong independence is, and it is what a reader would compute by hand.
The direct competitor is the standard i.i.d. out-of-bag interval of Wang, Wu and Nettleton, computed with the same forest, features and tree count. It is the natural zero-cost choice and the one with theory behind it. Comparing it with our method is also the key ablation: the only difference is block resampling with $L$ matched to $|W|$. We will sweep $L$ from one day to beyond $|W|$ to show that coverage tracks the block length rather than some other choice. That paper reports no numeric coverage values in the file, only that out-of-bag errors match test errors on Concrete, Airfoil, Bioconcentration and Naval.
The strongest time-series method is EnbPI (Xu and Xie). It builds bootstrap ensembles, uses leave-one-out residuals, and updates the interval with a sliding window of recent residuals. It reports coverage "around 0.9" across datasets including Beijing air quality. We will run it with the same base forest and the same number of bootstrap members. Its window size will be tuned on 2016–2018 placebo windows with the same budget we give our block length.
The strongest temporal-holdout competitor is split conformal prediction. The forest is trained on years before a held-out calibration year, calibrated on that year's window-mean residuals, and then refit or kept fixed. We tune it on the same budget.
For the point estimate itself, we include the transferable generalized additive model of Transferable Models to Understand the Impact of Lockdown Measures. That model freezes pre-lockdown weather relationships and reports NO₂ changes of −29.4% (Eastern Switzerland), −28.1% (Beijing) and −52.8% (Wuhan). No reported interval coverage for those estimates was found on Lacuna. We will attach residual-block intervals to that model as well. This checks whether the forest's effect estimates are even needed or whether the gain comes purely from the uncertainty method.
Beating the baselines means the following. Pooled placebo coverage of zero must fall within two binomial standard errors of 90%, where i.i.d. out-of-bag falls clearly below. Median interval width must be at most that of temporal-holdout split conformal at matched coverage. The kernel conditional-coverage test must not reject at 5% across weather regimes.
Why not just…?
Why not just use temporal-holdout split conformal and skip out-of-bag entirely? This is the most serious threat. Holding out the most recent year respects time order and carries a finite-sample guarantee if the calibration year resembles the target. Its weakness is concrete. Either the forest is trained without the year closest to 2020, or the calibration year is consumed and lost to training. Air quality in Beijing and Wuhan was declining year on year under clean-air policies, so a stale forest builds a trend bias into the counterfactual. That bias then appears as a spurious lockdown effect. Gupta and Ramdas show the same data-efficiency cost of splitting in a different setting: their non-split binning needs roughly an order of magnitude fewer samples than the split version. The placebo experiment settles the question. If split conformal matches block out-of-bag on coverage and width across 2017–2019 placebo years, we will recommend it and report block out-of-bag only as an equally good alternative that needs no refitting.
Why not just use EnbPI, which already handles time series? EnbPI's adaptivity is the wrong property for counterfactual estimation. Its sliding window feeds recent residuals back into the interval. Once the lockdown begins, those residuals contain the lockdown effect itself, so the interval widens to absorb the very signal we want to measure. Without updates, EnbPI reduces to i.i.d. leave-one-out residuals, which share the interpolation problem. Its reported ~0.9 coverage on Beijing air quality concerns one-step-ahead hourly prediction, not multi-week aggregate counterfactuals. We will run it both with and without updates on placebo windows and on 2020. If its updated variant shrinks the estimated lockdown effect toward zero relative to all other methods, that confirms the contamination.
Why not just check calibration with coverage by concentration decile, or a histogram of where outcomes fall within the predictive distribution? These are standard and easy to read. If they give the same verdict under any reasonable binning, the kernel test adds little, and we will say so. The concern is that placebo samples are small: a few hundred overlapping windows per city. In that regime the binning literature predicts both failure modes. Wide bins mask opposite errors in clean and haze regimes (Kumar et al.), and narrow bins produce noisy positive "miscalibration" (Widmann et al.). We will report the binned verdict for bin counts from 3 to 20, under both equal-width and equal-mass bins, next to the kernel test. The claim is only that binned verdicts flip across these choices on our data while the kernel verdict does not. If they do not flip, the binning-free part becomes a robustness check rather than a contribution.
Proposed Experiments
The primary dataset is the multi-region lockdown data of Transferable Models. It covers NO₂, PM10 and PM2.5 with weather and mobility proxies for Eastern Switzerland, Lower Austria, Beijing and Wuhan from January 2016 to February 2021. That span supplies true placebo windows, namely the 23 January–27 April window in 2017, 2018 and 2019. It also supplies rolling 2-, 4- and 8-week windows throughout 2016–2019, each with the forest trained only on data before the window. The pooled placebo set spans four regions, three pollutants and three window lengths. We will report coverage of zero, median width, and the kernel conditional-coverage test over weather regime (stagnation versus ventilated days) and concentration level.
The decisive check is a gap. i.i.d. out-of-bag should under-cover on placebo windows, increasingly so as $|W|$ grows. Block out-of-bag should not. If i.i.d. out-of-bag already achieves nominal placebo coverage, the core premise is false and the proposal reduces to recommending existing intervals.
The secondary check applies the calibrated method to 2020–2021. We will recompute each city's lockdown effect and recovery date and report how much the conclusions move. In particular we want to know whether Eastern Switzerland's +9.8% PM10, attributed to a Sahara dust event, falls inside a zero-effect interval. That would separate weather confounding from model error, a separation the original study could not make.
Two supporting experiments test generality. The Beijing hourly air-quality series used by Xu and Xie contains no lockdown, so it serves purely as a placebo benchmark with many more windows. On it we can compare EnbPI's own reported setting with the aggregate-effect setting. The Yangtze River Delta data of Lu et al. covers only 2019-I and 2020-I, so it tests a spatial rather than temporal version of the problem. There, blocks are spatial tiles, and we check whether the 3.61 µg/m³ regional drop survives intervals that respect spatial autocorrelation.
All methods share the same forest hyperparameters, chosen once on 2016 data following Probst et al.. All use the same number of trees and the same tuning budget for their one free parameter: block length, EnbPI window, or split-conformal calibration year.
References
- AirDelhi: Fine-Grained Spatio-Temporal Particulate Matter Dataset From Delhi For ML based Modeling
- Ambient PM2.5 Estimates and Variations during COVID-19 Pandemic in the Yangtze River Delta Using Machine Learning and Big Data
- Stability of Random Forests and Coverage of Random-Forest Prediction Intervals
- Global fossil carbon emissions rebound near pre-COVID-19 levels
- Verified Uncertainty Calibration
- Calibration tests in multi-class classification: A unifying framework
- Distribution-Free Calibration Guarantees for Histogram Binning without Sample Splitting
- Conformal Prediction for Time Series
- Transferable Models to Understand the Impact of Lockdown Measures on Local Air Quality
- Hyperparameters and tuning strategies for random forest