Reflection on modern methods: constructing directed acyclic graphs (DAGs) with domain experts for health services research

Daniela Rodrigues, Noémi Kreif, Anna Lawrence‐Jones, Mauricio Barahona, Erik MayerInternational Journal of Epidemiology 2022 · DOI

TL;DR

This paper addresses the challenge of constructing Directed Acyclic Graphs (DAGs) for health services research, particularly when dealing with complex interventions and limited existing literature. It proposes and reflects on a practical framework for co-creating DAGs with domain experts, including patients and healthcare professionals, to ensure more robust and relevant causal models.

What Problem Does This Paper Address?

In health services research, understanding "what works" often requires estimating the causal effect of interventions. While Directed Acyclic Graphs (DAGs) are powerful tools for visualizing causal assumptions and identifying confounders, their adoption in applied health research is hindered by a lack of practical guidance. This is especially true when evaluating novel or complex interventions where existing literature is scarce. Traditional approaches often rely solely on researchers' interpretation of literature, potentially missing crucial real-world nuances. This paper tackles the gap in how to effectively build DAGs by incorporating the invaluable knowledge of domain experts, moving beyond literature-based construction to a more collaborative and grounded approach.

Methods

The paper describes a four-phase framework for DAG development with domain experts:

  1. Brainstorming: Initial creation of a DAG with multiple potential outcomes, focusing on identifying the intervention, its expected outcomes, and key factors influencing intervention assignment.
  2. Refinement: Focusing the DAG on a specific outcome, iteratively refining the graph by omitting arrows (assuming no causal relationship) and identifying common causes of the intervention and outcome.
  3. Exposition: Presenting literature-driven DAGs to participants for feedback and discussion.
  4. Reconciliation: Analyzing and combining features from all proposed DAGs (expert- and literature-driven) into a final set, and then applying the backdoor criterion to identify a minimal adjustment set for confounding.

The authors conducted a stakeholder workshop involving patients, healthcare professionals, researchers, commissioners, and industry representatives to develop DAGs for an online consultation intervention in the English National Health Service. They used Microsoft PowerPoint for initial DAG creation but recommend specialized software like DAGitty or Fusion for future applications.

Datasets

Not discussed in the paper. The paper focuses on the process of DAG construction rather than analyzing specific datasets.

Key Insights and Ideas

  • The Power of Collaboration: Involving domain experts (patients, clinicians, etc.) in DAG construction leads to more robust, realistic, and relevant causal models than relying solely on researchers' literature reviews. This is particularly crucial for complex interventions or when published evidence is limited.
  • Bridging the Gap: DAGs offer a structured way to translate qualitative causal assumptions into a format that can guide quantitative analysis, making the underlying reasoning explicit and transparent.
  • Practical Challenges: Building DAGs is not always straightforward. Challenges include defining variables precisely, ensuring they are measurable, and achieving consensus among diverse stakeholders, which can lead to multiple DAGs representing a single relationship.
  • The Backdoor Criterion is Key: This criterion provides a systematic way to identify the minimal set of variables needed to control for confounding, thereby strengthening the credibility of causal inference from observational data.
  • Iterative Refinement is Essential: The process of building a DAG is iterative, involving brainstorming, refinement, and reconciliation to ensure all relevant factors are considered and assumptions are clearly articulated.

Key Results

The paper doesn't present quantitative results in terms of statistical model performance. Instead, its key results are qualitative and methodological:

  • Successful Stakeholder Engagement: A workshop successfully brought together diverse domain experts to collaboratively construct DAGs for a complex intervention (online consultation).
  • Identification of Confounders: The process led to the identification of a comprehensive set of confounding variables for the intervention, including those identified from literature and those unique to expert knowledge. For the example intervention, 15 confounding variables were identified as a minimal adjustment set.
  • Practical Guidance: The paper provides a framework and lessons learned for researchers interested in engaging domain experts to develop DAGs, highlighting the benefits and challenges of this approach.

Limitations and Future Work

  • Time Constraints: The workshop format, while effective, faced time limitations, preventing a full discussion of literature-driven DAGs with participants during the event. Sharing these via email resulted in limited feedback.
  • Facilitator Dependency: In the described workshop, facilitators played a significant role in drawing the DAGs, suggesting that future sessions might benefit from facilitators with strong DAG expertise to guide the drawing process more actively.
  • Software Recommendations: While PowerPoint was used, the authors recommend specialized DAG software (DAGitty, Fusion) for more efficient and structured DAG creation.
  • Future Directions: The authors suggest future work could focus on discussing other assumptions necessary for causal inference (e.g., positivity, consistency, selection bias, measurement bias, time-dependent confounding) and exploring how to handle unmeasured confounding through sensitivity analyses or partial identification strategies. They also emphasize the value of multi-session workshops to enhance the robustness and relevance of causal models.

Summary updated

Abstract

Directed acyclic graphs (DAGs) are a useful tool to represent, in a graphical format, researchers' assumptions about the causal structure among variables while providing a rationale for the choice of confounding variables to adjust for. With origins in the field of probabilistic graphical modelling, DAGs are yet to be widely adopted in applied health research, where causal assumptions are frequently made for the purpose of evaluating health services initiatives. In this context, there is still limited practical guidance on how to construct and use DAGs. Some progress has recently been made in terms of building DAGs based on studies from the literature, but an area that has received less attention is how to create DAGs from information provided by domain experts, an approach of particular importance when there is limited published information about the intervention under study. This approach offers the opportunity for findings to be more robust and relevant to patients, carers and the public, and more likely to inform policy and clinical practice. This article draws lessons from a stakeholder workshop involving patients, health care professionals, researchers, commissioners and representatives from industry, whose objective was to draw DAGs for a complex intervention-online consultation, i.e. written exchange between the patient and health care professional using an online system-in the context of the English National Health Service. We provide some initial, practical guidance to those interested in engaging with domain experts to develop DAGs.
BibTeX
@article{rodrigues2022reflection,
  author = {Daniela Rodrigues and Noémi Kreif and Anna Lawrence‐Jones and Mauricio Barahona and Erik Mayer},
  title = {{Reflection on modern methods: constructing directed acyclic graphs (DAGs) with domain experts for health services research}},
  journal = {International Journal of Epidemiology},
  year = {2022},
  doi = {10.1093/ije/dyac135},
}
Figure