The reporting of study limitations and hedging in articles with and without causal diagrams

Taísa Rodrigues Cortes, Ismael Henrique da Silveira, Michael Maia Schlüssel, Guilherme Loureiro Werneck, Claúdio J. StruchinerResearch Square 2023 · DOI

Abstract

Causal directed acyclic graphs (DAG) are known to constitute a useful tool to improve the methodological and reporting quality of observational studies. At the same time, there are concerns that their (mis)use may lead to overconfident inferences. In this study, we investigate the reporting of study bias and limitations, and the use of hedging in articles with and without causal DAG. We performed a keyword search in two electronic databases (Medline and Web of Science) and one full-text repository (PubMed Central). The search strategy aimed to identify biomedical studies with an analytical observational design, that reported use of causal DAG. We selected a random sample of the articles that presented a graphic structure. We pair-matched individually 130 DAG with 130-Non-DAG papers on journal and issue. The number of research limitations acknowledged by the authors and of hedging terms per 100 words were analyzed using multiple linear regression models. From 2000 to 2020, we identified 2256 publications reporting the use of causal DAG, of which 1131 (50%) contained the actual graph structure. The number of study limitations acknowledged by the authors and the frequency of hedging were found to be similar in both samples. However, quantitative evaluations of study limitations were more frequently reported in studies with DAG (51 % vs. 25 %), while bias modeling techniques were uncommon in both groups (18% in DAG and 6% in non-DAG). In order to contextualize our findings, we used causal diagrams to discuss potential mechanisms for the communication of causal assumptions and related uncertainties in observational studies.

Introduction

In observational research, identifying cause-and-effect relationships is notoriously difficult due to confounding factors and selection biases. Causal Directed Acyclic Graphs (DAGs) have emerged as a powerful tool to visually map out these relationships and clarify underlying assumptions. However, a debate exists within the scientific community: do DAGs improve transparency, or do they provide a "veneer of rigor" that leads to overconfident conclusions? Critics worry that formalizing assumptions might make researchers less likely to acknowledge the inherent uncertainties of their data. This study investigates this tension by comparing how authors of biomedical articles with and without DAGs report study limitations and use "hedging"—linguistic cues like "suggests" or "possibly" that signal caution.

Methodology

The researchers conducted a comparative study using a matched-pair design to control for "journal culture." They searched databases including Medline, Web of Science, and PubMed Central for biomedical observational studies published between 2000 and 2020 that utilized DAGs. From an initial pool of over 13,000 records, the team narrowed the selection down to a final sample of 260 articles (130 pairs), matching each DAG-containing paper with a non-DAG paper from the same journal and issue.

Flowchart of the article selection process, illustrating the path from 13,103 identified records to the final sample of 260 articles.

Linguistic analysis was performed using the AntConc tool to calculate the frequency of hedging terms per 100 words in the discussion sections. Additionally, a manual review classified how limitations were reported: whether they were merely acknowledged, qualitatively discussed, or quantitatively assessed (e.g., via sensitivity analysis).

Results and Key Insights

The study found no evidence that DAGs lead to overconfidence. Authors using DAGs employed hedging terms at nearly the same rate as those who did not (4.0 vs 3.8 per 100 words). The most striking difference was not in the volume of limitations discussed, but in the depth of the evaluation. DAG users were significantly more likely to move beyond simple acknowledgment toward quantitative evaluation.

Regression coefficients for the number of limitations acknowledged and hedging frequency, showing negligible differences between DAG and non-DAG groups.

Key findings include:

  • Quantitative Assessment: 51% of DAG-based articles included a quantitative assessment of limitations (such as sensitivity analysis), compared to only 25% of non-DAG articles.
  • Odds Ratio: After matching and adjusting for confounders, the use of a DAG was associated with a 4.08 times higher odds of reporting a quantitative assessment of bias.
  • Precision of Language: DAG articles used more explicit, technical language to describe biases (e.g., "recall bias," "confounding"), suggesting that the process of drawing a DAG provides researchers with a more precise vocabulary for discussing potential pitfalls.

Discussion and Future Work

Despite the benefits of DAGs in encouraging quantitative assessment, advanced "bias modeling" remains rare in both groups, appearing in only 18% of DAG papers and 6% of non-DAG papers. The authors acknowledge limitations in their own study, including the subjectivity of manual classification and the potential for automated tools to miss nuanced expressions of uncertainty.

Future meta-research should utilize advanced Natural Language Processing (NLP) to better capture the complexities of scientific communication. Additionally, the researchers call for more empirical evidence on how these causal tools are used in practice to help develop more robust reporting guidelines for observational studies.

Summary updated

Abstract

<title>Abstract</title> Causal directed acyclic graphs (DAG) are known to constitute a useful tool to improve the methodological and reporting quality of observational studies. At the same time, there are concerns that their (mis)use may lead to overconfident inferences. In this study, we investigate the reporting of study bias and limitations, and the use of hedging in articles with and without causal DAG. We performed a keyword search in two electronic databases (Medline and Web of Science) and one full-text repository (PubMed Central). The search strategy aimed to identify biomedical studies with an analytical observational design, that reported use of causal DAG. We selected a random sample of the articles that presented a graphic structure. We pair-matched individually 130 DAG with 130-Non-DAG papers on journal and issue. The number of research limitations acknowledged by the authors and of hedging terms per 100 words were analyzed using multiple linear regression models. From 2000 to 2020, we identified 2256 publications reporting the use of causal DAG, of which 1131 (50%) contained the actual graph structure. The number of study limitations acknowledged by the authors and the frequency of hedging were found to be similar in both samples. However, quantitative evaluations of study limitations were more frequently reported in studies with DAG (51 % vs. 25 %), while bias modeling techniques were uncommon in both groups (18% in DAG and 6% in non-DAG). In order to contextualize our findings, we used causal diagrams to discuss potential mechanisms for the communication of causal assumptions and related uncertainties in observational studies.
BibTeX
@inproceedings{cortes2023reporting,
  author = {Taísa Rodrigues Cortes and Ismael Henrique da Silveira and Michael Maia Schlüssel and Guilherme Loureiro Werneck and Claúdio J. Struchiner},
  title = {{The reporting of study limitations and hedging in articles with and without causal diagrams}},
  booktitle = {Research Square},
  year = {2023},
  doi = {10.21203/rs.3.rs-2557666/v1},
}
Figure