Sensitivity analysis of the effects of non-pharmaceutical interventions on COVID-19 in Europe

Kristian Soltesz, Fredrik Gustafsson, Toomas Timpka, Joakim Jaldén, Carl Jidling, Albin Heimerson, Thomas B. Schön, Armin Sprečo, Joakim Ekberg, Örjan Dahlström, Fredrik Bagge Carlson, Anna Jöud, Bo BernhardssonmedRxiv 2020 · DOI

Abstract

The role of non-pharmaceutical interventions (NPIs) on the spread of SARS-CoV-2 has drawn significant attention, both scientific and political. Particularly, an article by the Imperial College COVID-19 Response Team (ICCRT), published online in Nature on June 8, 2020, evaluates the efficiency of 5 NPIs. Based on mortality data up to early May, it concludes that only one of the interventions, lockdown, has been efficient in 10 out of 11 studied European countries. We show, via simulations using the ICCRT model code, that conclusions regarding the effectiveness of individual NPIs are not justified. Our analysis focuses on the 11th country, Sweden, an outlier in that no lockdown was effectuated. The new simulations show that estimated NPI efficiencies across all 11 countries change drastically unless the model is adapted to give the Swedish data special treatment. While stated otherwise in the Nature article, such adaptation has been done in the model code reproducing its results: An ungrounded country-specific parameter said to have been introduced in all 11 countries, is in the code only activated for Sweden. This parameter de facto provides a new NPI category, only present in Sweden, and with an impact comparable to that of a lockdown. While the considered NPIs have unarguably contributed to reduce virus spread, our analysis reveals that their individual efficiency cannot be reliably quantified by the ICCRT model, provided mortality data up to early May.

The Problem of Identifiability in NPI Modeling

In June 2020, the Imperial College COVID-19 Response Team (ICCRT) published a high-profile study in Nature claiming that among five non-pharmaceutical interventions (NPIs), only full lockdowns had a significant impact on reducing the virus's reproduction number ($R_t$) across 11 European countries.

This paper identifies a major "identifiability" problem: because most countries implemented several NPIs—such as school closures, public event bans, and social distancing—within just a few days of each other, it is mathematically difficult to determine which specific measure caused the decline in deaths. The authors specifically scrutinize the model's performance regarding Sweden, the only country in the set that did not implement a lockdown, to determine if the model's conclusions are robust or rely on "special treatment" to fit the data.

Methodology and Data Sensitivity

The researchers performed an empirical sensitivity analysis using the original Bayesian estimation model code and data shared by the ICCRT. The study utilizes mortality data from the European Centre for Disease Prevention and Control (ECDC) and NPI timelines for 11 countries: Austria, Belgium, Denmark, France, Germany, Italy, Norway, Spain, Sweden, Switzerland, and the United Kingdom.

Timelines of NPIs across 11 European countries, highlighting the specific case of Sweden and the perturbation of intervention dates.

The analysis involved:

  • Parameter Toggling: Investigating a specific "bonus" parameter for the last intervention introduced in a country.
  • Data Perturbation: Testing model reactions to slight shifts in NPI dates (e.g., changing the definition of when "school closures" began in Sweden).
  • Data Pooling Alteration: Running the model on subsets of countries (specifically the UK and Sweden) to see if the "lockdown" effect remained dominant when data was pooled differently.

Key Insights: The "Sweden Fix" and "Last Man Standing" Bias

The most striking finding is that the ICCRT code included a country-specific parameter for the "last intervention" that was only activated for Sweden. While the Nature article suggested this was a general model feature, the code revealed it was a "virtual NPI category" used only to make the Swedish data fit. Without this fix, the model could not explain why Sweden’s $R_t$ dropped without a lockdown.

Furthermore, the model exhibits a "Last Man Standing" bias, where it tends to attribute almost all success to whichever intervention happened to be implemented last. In most countries, this was lockdown; in Sweden, it was the public events ban. Subtle changes in definitions (e.g., whether "school closure" in Sweden meant universities or primary schools) caused the model to radically reassign the "credit" for reducing $R_t$ from one NPI to another.

Results: Fragility of the Lockdown Narrative

When the "last intervention fix" was disabled for Sweden, the estimated importance of the "public events ban" increased significantly across all 11 countries. In scenarios where only the UK and Sweden were pooled and the "fix" was disabled, the model suddenly attributed the reduction in $R_t$ to the public events ban, estimating the effect of "lockdown" to be negligible.

Estimated relative reduction in $R_t$ for different governmental interventions, showing the dominance of lockdown in the original model's baseline results.

Numerical instability was also observed; when the authors attempted to apply the "last intervention" parameter to all countries (as the original paper claimed to do), the statistical engine indicated the model was over-parameterized and the results were unreliable.

Conclusion

While NPIs undoubtedly helped reduce the spread of the virus, this specific model cannot reliably quantify which NPI worked best. The individual efficiencies of these measures are not identifiable given the close proximity of their implementation dates. The authors conclude that simplistic, data-driven phenomenological models can provide a false sense of certainty and that rigorous sensitivity analysis is essential before such models are used for policymaking.

Summary updated

Abstract

Abstract The role of non-pharmaceutical interventions (NPIs) on the spread of SARS-CoV-2 has drawn significant attention, both scientific and political. Particularly, an article by the Imperial College COVID-19 Response Team (ICCRT), published online in Nature on June 8, 2020, evaluates the efficiency of 5 NPIs. Based on mortality data up to early May, it concludes that only one of the interventions, lockdown, has been efficient in 10 out of 11 studied European countries. We show, via simulations using the ICCRT model code, that conclusions regarding the effectiveness of individual NPIs are not justified. Our analysis focuses on the 11th country, Sweden, an outlier in that no lockdown was effectuated. The new simulations show that estimated NPI efficiencies across all 11 countries change drastically unless the model is adapted to give the Swedish data special treatment. While stated otherwise in the Nature article, such adaptation has been done in the model code reproducing its results: An ungrounded country-specific parameter said to have been introduced in all 11 countries, is in the code only activated for Sweden. This parameter de facto provides a new NPI category, only present in Sweden, and with an impact comparable to that of a lockdown. While the considered NPIs have unarguably contributed to reduce virus spread, our analysis reveals that their individual efficiency cannot be reliably quantified by the ICCRT model, provided mortality data up to early May.
BibTeX
@misc{soltesz2020sensitivity,
  author = {Kristian Soltesz and Fredrik Gustafsson and Toomas Timpka and Joakim Jaldén and Carl Jidling and Albin Heimerson and Thomas B. Schön and Armin Sprečo and Joakim Ekberg and Örjan Dahlström and Fredrik Bagge Carlson and Anna Jöud and Bo Bernhardsson},
  title = {{Sensitivity analysis of the effects of non-pharmaceutical interventions on COVID-19 in Europe}},
  year = {2020},
  doi = {10.1101/2020.06.15.20131953},
  howpublished = {medRxiv},
}
Figure