Reply to: The effect of interventions on COVID-19

Seth Flaxman, Swapnil Mishra, James A. Scott, Neil M. Ferguson, Axel Gandy, Samir BhattNature 2020 · DOI · OpenAlex

In mid-2020, a landmark study by Flaxman et al. concluded that "lockdown" was the only non-pharmaceutical intervention (NPI) that significantly reduced the reproduction number ($R_t$) of COVID-19 across 11 European countries. This finding sparked a high-profile exchange in Nature with Soltesz et al., who challenged the statistical methods used to evaluate these interventions.

The Challenge of Identifiability

The primary point of contention is identifiability. When governments implement multiple measures—such as banning public events, closing schools, and mandating lockdowns—within a very short timeframe, statistical models struggle to isolate the impact of any single policy. Soltesz et al. argue that the original model's structure was overly flexible, leading it to "credit" the last intervention implemented in a country with the majority of the success.

The Sweden Paradox

Sweden provided a unique case study as the only country in the original set that did not implement a full lockdown. Soltesz et al. demonstrated that the model shifted the credit for Sweden's declining $R_t$ to its last implemented NPI: a ban on public events. While the model attributed a ~71% reduction to this ban in Sweden, it attributed less than 2% to the same measure in the other 10 countries. This inconsistency suggests the model might be over-fitting to the final drop in deaths rather than identifying true causal links.

Flaxman et al.'s Defense: Partial Pooling

Flaxman et al. defended their findings by highlighting their Bayesian hierarchical modeling approach, specifically "partial pooling." Instead of treating each country in isolation or assuming identical effects everywhere, partial pooling allows for a shared global effect while accounting for country-specific variations. They argue that the joint analysis of 11 countries provides enough "statistical leverage" to identify a consistent pattern where lockdown remains the most significant NPI.

Estimated $R_t$ and relative reduction in transmission for various NPIs under the primary model (a). Lockdown shows a dominant effect of approximately 80%.

Robustness and Sensitivity

To address the critics' concerns, Flaxman et al. conducted various sensitivity tests. They showed that even if Sweden is excluded from the analysis, the estimated effect of lockdown remains high (approx. 81% reduction in $R_t$). Furthermore, simulation studies using synthetic data suggested the model could correctly recover known effect sizes even in the presence of unobserved variables.

Sensitivity analysis (b) showing how shifting model assumptions can alter the estimated effectiveness of individual NPIs, illustrating the identifiability debate.

Conclusion and Future Directions

Both parties acknowledge the inherent difficulty of using mortality data—a lagging indicator—to estimate transmission dynamics. The models struggle to account for voluntary social distancing that often precedes government mandates. Future work requires more granular, standardized data on policy implementation and potentially the inclusion of additional covariates like mobility data to provide a more complete picture of pandemic trajectories.

This summary is based on the "Matters Arising" exchange between Soltesz et al. and Flaxman et al. in Nature (2020).

Summary updated

BibTeX
@inproceedings{flaxman2020reply,
  author = {Seth Flaxman and Swapnil Mishra and James A. Scott and Neil M. Ferguson and Axel Gandy and Samir Bhatt and Seth R. Flaxman},
  title = {{Reply to: The effect of interventions on COVID-19}},
  booktitle = {Nature},
  year = {2020},
  doi = {10.1038/s41586-020-3026-x},
}
Figure