Known-Answer Tests for Satellite Evaluations of Ecological Restoration Policy

Many published studies of restoration policy in China describe an "ecological U-curve." Vegetation declines while mining or farming goes unchecked, then recovers after a policy is enacted, and the turn is attributed to the policy. The evidence usually comes from decades of satellite images, trend tests, and correlations. These studies rest on observational records collected after the fact, and the records have three properties that can manufacture or inflate a recovery signal. First, sites are often chosen for intervention because they have just hit a low point, and a site at a low point tends to rebound anyway. Second, policies roll out in waves, and simple regressions are known to mishandle staggered timing. Third, cloud cover removes satellite observations in ways that depend on season and terrain. Researchers can sensibly argue for different estimators, but nobody can currently say which one is right for these data, because no evaluation has a known true answer.

We propose to build one. We will take real multi-decade Landsat panels from the case regions in the literature, keep their real cloud gaps, real climate variability, and real spatial structure, and inject restoration effects of known size. The injected treatment timing will mimic how policies actually select sites. We then ask which estimators recover the known effect, under which observation windows, and how badly the methods used in published work fail. The same procedure then tells us which real-data estimates of recovery, such as those for the Taojiang and Shaoyang coal mines, can be trusted.

Core Claim

Our hypothesis is that, in retrospective satellite evaluations of restoration policy, the largest source of error is not cloud noise or estimator choice in isolation. It is the interaction between selection on a pre-policy dip and a truncated observation window. When a study window starts near the trough, or has too few pre-policy years to show that the trough was temporary, every estimator that relies on pre-period trends will credit the policy with recovery that would have happened anyway. Two kinds of rebound are at issue: regression toward the mean, and spontaneous regrowth after extraction stops.

Staggered rollout and cloud-driven missingness add error on top of that, and they matter mainly when pre-periods are short. If the hypothesis holds, the practical rule is about design rather than modelling. Analysts would need a minimum pre-policy window length and an explicit test for a pre-policy dip before any estimate is reported. That would change how the U-curve literature should be read.

The idea is new in that it supplies a known-answer evaluation for this question. Methods papers test estimators on economic panels. Synthetic Difference-in-Differences and Robust Synthetic Control both use California's 1988 tobacco tax panel. Ecology papers report effects without counterfactual checks. Neither group has asked how these estimators behave on vegetation panels with dip-based site selection and satellite missingness that depends on season and terrain.

How Prior Work Positions This

Three ecological studies motivate the proposal by producing the claims we want to test.

Policy-Driven Mine Ecological Restoration Projects in China tracks stone coal mines in Taojiang County from 2004 to 2023. Vegetation cover in the mining area fell to 0.432 in 2011 and recovered to 0.718 by 2023 after regulations enacted in 2011–2013. The study uses trend tests and Random Forest rankings rather than any counterfactual. It also admits that cloud cover forced it to mix calendar months across years.

Drivers of Vegetation Cover and Carbon Sink Dynamics in Abandoned Shaoyang City Open-Pit Coal Mines reports the same U-shape for 1998–2024. It concludes that policy overrides climate because vegetation cover correlates only weakly with rainfall (r = 0.18) and temperature (r = 0.41). Weak correlations of this kind cannot rule out a post-disturbance rebound.

Exploring the Impact of Grain-for-Green Program on Trade-Offs and Synergies among Ecosystem Services in West Liao River Basin, China goes furthest toward causal inference. It combines propensity score matching (pairing treated and untreated subbasins on recorded baseline traits) with difference-in-differences and finds a +1.3% gain in carbon sequestration. It still treats the program as a single shock in 2000 and works from four decadal snapshots, which is a clear case of temporal truncation.

Two papers constrain the design.

Choosing an analytic approach: key study design considerations in state policy evaluation shows that standard two-way fixed effects regression is biased under staggered adoption. A two-way fixed effects regression has one dummy per unit and one per year; under staggered adoption it uses already-treated units as controls for later-treated units. On a naloxone-law panel, the paper finds that reasonable estimators disagree in sign, ranging from −1.32 to +2.32. That result is the reason we need ground truth: the paper shows disagreement but has no means to settle it.

The Impact of National Ecological Civilization Pilot Zone Policy on Rural Residents' Income—Evidence from Chinese Counties supplies a real staggered rollout in 2016, 2017, and 2019 across 1,761 counties, and its panel includes NDVI. It analyzes only income (a +2.37% effect), so the ecological outcome in that panel has not been evaluated.

Three methods papers supply the techniques we reuse.

Synthetic difference-in-differences reweights control units to match the treated unit's trend and reweights pre-policy years to resemble post-policy years. It is designed for a single adoption date.

Robust synthetic control fills in and de-noises a panel by low-rank matrix completion before fitting donor weights. Its pre-period error bound shrinks as $O(1/\sqrt{T_0})$ in the pre-period length $T_0$. That bound is the formal reason pre-window length should matter.

Time-Aware Synthetic Control adds a state-space trend model fitted by Kalman smoothing, because classical synthetic control ignores the order of pre-policy years.

None of the three has been tested on satellite vegetation panels with dip-based site selection. That gap is what we evaluate.

Research Program

The core artifact is a semi-synthetic testbed built on real data.

For each case region, we will assemble annual Landsat 5, 8, and 9 panels at 30 m resolution, aggregated to grid cells of roughly 1 km. The regions are Shaoyang 1998–2024, Taojiang 2004–2023, and West Liao 1990–2020 using annual rather than decadal data. Each panel carries the real per-scene cloud and quality masks, gridded climate, elevation, and road distance. We will choose donor cells that, as far as can be checked, received no restoration intervention.

On these cells we inject a known outcome trajectory:

$$ Y_{it} = Y^{0}_{it} - D_i\, g(t - s_i) + R_i\, h(t - c_i) + \tau\, \mathbf{1}[t \ge a_i]\, k(t - a_i). $$

The terms are defined as follows. $Y_{it}$ is vegetation cover in cell $i$ and year $t$. $Y^{0}_{it}$ is the real observed series for an untouched donor cell. $D_i\, g(\cdot)$ is an injected extraction dip that starts in year $s_i$. $R_i\, h(\cdot)$ is spontaneous regrowth that begins once extraction ceases in year $c_i$. $\tau\, k(\cdot)$ is the true policy effect, with its own ramp-up shape, starting at adoption year $a_i$.

Adoption years are assigned the way policies plausibly assign them. They come in staggered cohorts, and the probability of adoption rises with the depth of the recent dip, with a tunable strength of selection. Observations are then removed using the real cloud masks of the same cells, so the missingness pattern really is seasonal and terrain-driven. In a second condition, we also correlate missingness with the peak growing season. Finally, each panel is cropped to windows of varying pre-policy length, from 2 to 13 years.

This gives a grid of scenarios with known true effects. The axes are selection strength, the size of spontaneous regrowth (including zero), staggering, missingness mechanism, and window length.

The proposal's own estimator is deliberately modest. It is synthetic difference-in-differences run separately for each adoption cohort, using only not-yet-treated and never-treated controls, and averaged across cohorts. Two additions address the hypothesized failure modes. The first is matrix completion of cloud gaps in the style of robust synthetic control, applied to deseasonalized annual composites. The second, which we call a dip-aware pre-window, excludes the years immediately before adoption from the trend-matching step and requires donors to match the pre-dip trajectory. The goal is not to claim the best estimator. It is to identify which ingredient removes which bias, and how much pre-window is needed before any estimator is reliable.

Baselines

The naive reference point is the published practice itself: Theil–Sen slopes and Mann–Kendall break tests, comparing the post-policy slope with the pre-policy slope, as in the Taojiang and Shaoyang studies. We read the implied "policy effect" as the post-policy level minus the pre-policy level, and we run this on exactly the same semi-synthetic panels. Any method that cannot beat this has no claim to be used.

The second baseline is two-way fixed effects difference-in-differences with annual climate covariates. This is the standard econometric default and the model reviewers will reach for first. We run it on identical panels with listwise deletion of cloud-missing years.

The third is propensity score matching plus difference-in-differences, matched on elevation, slope, road distance, and baseline vegetation, as in the West Liao study. We run it at both decadal and annual resolution, so that the cost of the decadal snapshots can be isolated.

The strongest existing competitor is synthetic difference-in-differences, applied with block timing (the earliest adoption year) or cohort by cohort. It is the strongest because its unit and time weights already address trend mismatch, and the method is consistent if either set of weights balances the hidden structure. The only reported numbers for it come from the tobacco tax panel: an estimate of −15.6 with standard error 8.4, against −27.3 (standard error 17.7) for classical difference-in-differences and −19.6 (standard error 9.9) for synthetic control, all reported in Synthetic Difference-in-Differences. We found no reported result on Lacuna for synthetic difference-in-differences, time-aware synthetic control, or robust synthetic control on any remote-sensing vegetation panel.

Robust synthetic control and time-aware synthetic control serve as additional strong baselines. We run them cell by cell for single treated units and aggregate the results.

All estimators receive the same annual composites, the same donor pool, and the same tuning budget. Regularization strength and imputation rank are chosen only by placebo fit on pre-policy years, never by looking at the injected truth.

Ablations isolate each ingredient: our estimator without the dip-aware pre-window, without matrix completion, and without cohort separation. Beating the strongest baseline means two things together. Bias in the average effect on treated cells must be lower in the high-selection, short-window scenarios, and 95% interval coverage must stay near nominal. Any method that wins on bias by inflating its intervals does not count.

Why not just…?

Why not just build cloud-free annual maximum composites and run standard synthetic difference-in-differences? This is the most serious threat. Annual maximum compositing in Google Earth Engine removes much of the seasonal mismatch that the Taojiang study complains about, and it may make missingness nearly harmless. We include exactly this pipeline as a baseline. If it matches our estimator across the scenario grid, then the missingness component of the proposal is unnecessary, and the paper's finding becomes that dip selection and window length are what matter. We think that outcome is plausible and still useful. Compositing cannot remove the dip problem, however. A composite of a site selected because it crashed still crashed.

Why not just run several estimators on the real data and report how much they disagree? That is what Choosing an analytic approach did, and the outcome was an unresolved spread from −1.32 to +2.32. A spread tells you the answer is uncertain but not which answer is wrong. The injected-truth testbed exists so that disagreement on the real Shaoyang and Taojiang panels can be interpreted. Estimators that fail in the matching semi-synthetic scenario can be discounted.

Why not just add climate covariates to a fixed-effects regression or a Random Forest and read off the policy's contribution? Fixed-effects regression is biased under staggered timing for the reasons Stone and colleagues give. Random Forest importance rankings, such as road distance accounting for a 9.15% increase in mean squared error in the Taojiang study, measure predictive association rather than effect. Neither approach models the regrowth that follows when extraction stops. The testbed measures this failure directly: it reports how far each estimate lands from the injected effect when climate is the real climate and the effect size is known.

Proposed Experiments

The first experiment is the full scenario grid on the three regional panels. For each estimator we report bias, root mean squared error, 95% interval coverage, and sign-error rate. The headline output is a map of where each method fails. The decisive check for the core claim is whether the bias of trend-based methods under dip selection falls as pre-window length grows, and whether it falls roughly at the $1/\sqrt{T_0}$ rate suggested by the robust synthetic control bound. If bias is driven mainly by missingness or staggering even when pre-windows are long, the hypothesis is wrong.

The second experiment applies the methods to real data. On Shaoyang we place placebo policy years at 2003 and 2005, where no policy occurred, and run a pre-window sweep that varies the study start date from 1998 to 2008. We then compare the estimated policy effect against a regrowth control built from extraction zones where, according to the Shaoyang paper, enforcement lagged. If the recovery estimate shrinks sharply once the regrowth control and the longer pre-window are included, the published attribution to policy is overstated.

The third experiment uses the pilot-zone county panel and its real three-wave rollout to estimate the policy's effect on NDVI, which has not been estimated before. Here we compare cohort-separated synthetic difference-in-differences with two-way fixed effects to see whether the forbidden-comparison bias changes the conclusion.

A final sanity check reruns our pipeline on the tobacco tax panel. The method should reproduce estimates near the published synthetic difference-in-differences value of −15.6, which confirms that our implementation is faithful before any ecological conclusion is drawn.