Abstract. In this note, we argue that the outputs of causal discovery algorithms should not usually be considered end results but rather starting points and hypotheses for further study. The incentive to explore this topic came from a recent study by Krich et al. (2020), which gives a good introduction to estimating causal networks in biosphere–atmosphere interaction but confines the scope by investigating the outcome of a single algorithm. We aim to give a broader perspective to this study and to illustrate how not only different algorithms but also different initial states and prior information of possible causal model structures affect the outcome. We provide a proof-of-concept demonstration of how to incorporate expert domain knowledge with causal structure discovery and remark on how to detect and take into account over-fitting and concept drift.
The Problem: Algorithms as Opinionated Black Boxes
In geosciences, researchers use Causal Structure Discovery (CSD) algorithms to determine directional relationships in complex systems, such as how solar radiation affects net ecosystem exchange. However, relying on a single algorithm is risky. Different tools—like PC-stable (constraint-based), GES (score-based), and LiNGAM (non-Gaussianity-based)—rely on different underlying assumptions. As a result, they often produce wildly different causal graphs from the identical dataset.
A Human-in-the-Loop Methodology
The authors propose moving beyond "data-only" discovery toward a five-step Bayesian-inspired workflow that integrates expert domain knowledge:
- Prior Knowledge Input: Experts provide probabilities for known physical structures.
- Multi-Algorithm Application: Applying several distinct algorithms to identify areas of consensus and divergence.
- Model Selection: Users select a starting model based on statistical scores and scientific plausibility.
- Iterative Refinement: The model is refined through interaction (simulated here via a "greedy search" for small edits).
- Validation: Using $k$-fold cross-validation to detect over-fitting and ensure the model holds up across different data subsets.
Key Insights: The Stabilizing Power of Priors
Using both synthetic data and real-world flux tower data from the SMEAR II station in Finland, the study demonstrates that expert knowledge acts as a stabilizer. Without priors, algorithms are prone to finding patterns in noise or missing well-established physical links (e.g., radiation driving temperature).
As the "certainty" of prior knowledge ($k$) increases, the Structural Hamming Distance (SHD)—a measure of how much models differ from the ground truth or each other—drops sharply. This indicates that even a small amount of expert input significantly improves model reliability and consistency.
Results and Future Outlook
In the real-world case study, the expert-informed approach identified four direct causal links from solar radiation that automated algorithms missed. By reducing the SHD between the expert model and algorithm-derived models from 10 to 5 through iterative refinement, the authors show that a "human-in-the-loop" approach brings data-driven models much closer to physical reality.
The paper serves as a call to action for the development of practical software tools that allow geoscientists to interactively "nudge" these algorithms in real-time, treating expert knowledge as a first-class citizen in the discovery process.
Summary updated