Abstract
Causal machine learning (ML) algorithms recover graphical structures that tell us something about cause-and-effect relationships. The causal representation provided by these algorithms enables transparency and explainability, which is necessary for decision making in critical real-world problems. Yet, causal ML has had limited impact in practice compared to associational ML. This paper investigates the challenges of causal ML with application to COVID-19 UK pandemic data. We collate data from various public sources and investigate what the various structure learning algorithms learn from these data. We explore the impact of different data formats on algorithms spanning different classes of learning, and assess the results produced by each algorithm, and groups of algorithms, in terms of graphical structure, model dimensionality, sensitivity analysis, confounding variables, predictive and interventional inference. We use these results to highlight open problems in causal structure learning and directions for future research. To facilitate future work, we make all graphs, models, data sets, and source code publicly available online.
Introduction
While standard "associational" machine learning—predicting an outcome $Y$ given features $X$—has seen massive adoption, Causal Machine Learning (CML) remains largely confined to academic settings. CML aims to uncover the underlying "why" behind data, which is essential for high-stakes decision-making. In a pandemic context, for instance, policy makers need to know the effect of an intervention (e.g., "What happens if we mandate masks?") rather than just observing correlations between mask usage and infection rates.
However, most CML algorithms are developed and tested using "clean" synthetic data. Real-world data is inherently messy, characterized by missing values, noise, and "distribution shifts" where relationships change over time. This study provides a rigorous "stress test" for causal structure learning by applying 29 different algorithms to real-world UK COVID-19 pandemic data, revealing significant gaps between theoretical promise and practical application.
Methodology
The researchers conducted a comprehensive comparative study involving 64 experiments based on 29 different algorithms. These algorithms represent several major classes of causal structure learning:
- Constraint-based: Utilizing statistical independence tests to identify edges (e.g., PC-Stable, FCI).
- Score-based: Searching for the graph structure that maximizes a "fitness" score (e.g., Hill-Climbing, GOBNILP).
- Hybrid: Combining constraint and score-based approaches (e.g., MMHC, GFCI).
- Continuous Optimization: Treating graph discovery as a mathematical optimization problem (e.g., NOTEARS).
- Model-Averaging: A meta-approach where results from multiple algorithms are aggregated to find a consensus structure.
To evaluate these algorithms, the authors established a "Knowledge Graph" based on expert consensus to serve as a ground truth. Performance was measured using metrics such as Structural Hamming Distance (SHD), F1 scores, and interventional inference via Pearl’s do-operator.
Data Description
The study utilized a newly collated dataset covering 866 days of the UK COVID-19 pandemic (January 2020 – June 2022). The dataset comprises 17 variables spanning viral testing, infections, hospitalizations, mortality, mobility, and policy interventions.
To accommodate the varying requirements of different algorithms, the authors generated seven distinct data formats. These included raw continuous data, discretized versions (using quartiles or k-means clustering), and datasets with either imputed values or original missing data.
Analysis and Insights
The experiments yielded several critical insights into the current state of causal discovery:
- The "Hairball" Problem: Algorithms applied to continuous data frequently produced extremely dense graphs with billions of parameters. These models likely overfit to "associational noise" rather than capturing sparse, meaningful causal links.
- Predictive Accuracy vs. Causal Validity: Many models demonstrated high accuracy in cross-validation (e.g., predicting next-day case counts), yet their underlying causal structures were radically different and often contradictory. This highlights that a model can be an excellent "predictor" while failing as an "explainer."
- The Wisdom of the Crowds: Model-averaging emerged as one of the most robust techniques. By aggregating the outputs of multiple algorithms, the resulting graphs were typically sparser, more realistic, and aligned more closely with expert knowledge than any single algorithm.
- Temporal Blind Spots: Most standard causal algorithms assume "stationarity"—that relationships remain constant over time. Consequently, they struggled to correctly link hospitalizations to policy changes like lockdowns, failing to account for the inherent 2-3 week time lag.
Key Results
The study found a high degree of inconsistency across the board. Even within the same class of algorithms, discovered graphs varied significantly. SHD scores, which measure errors relative to the expert knowledge graph, ranged from 30 to 90. For comparison, a completely empty graph would have scored 37, indicating that many algorithms performed worse than a null model.
Furthermore, there was significant "interventional disagreement." When simulating hypothetical interventions, such as increasing testing capacity, some models predicted massive shifts in infection rates while others predicted zero effect. Such divergence underscores the danger of relying on a single causal model for public policy. Finally, the method of data preparation was found to be crucial; discretization via k-means clustering generally led to more stable and better-performing models than raw continuous values or simple quartiles.
Conclusion and Future Work
The authors identify several "open problems" that must be addressed for CML to become a reliable tool in practice:
- Non-Stationarity: Algorithms must be developed to handle time-series data where effects are delayed or where the underlying system dynamics change (e.g., due to new viral variants).
- Latent Confounders: The significant disagreement on "hidden" variables suggests a need for better validation methods for unobserved causes.
- Refined Model Averaging: Future research should explore weighted averaging techniques that prioritize algorithms based on their robustness to specific data characteristics.
To support further research, the authors have made all graphs, models, datasets, and source code publicly available online.
Summary updated