Centralized scientific communities are less likely to generate replicable results

Valentin Danchev, Andrey Rzhetsky, James A. EvanseLife 2019 · DOI

TL;DR

This study reveals that scientific communities that are highly centralized, relying on similar methods and shared authors, tend to produce less replicable results. Conversely, decentralized communities with diverse methods and independent teams are more likely to generate robust findings, suggesting that fostering decentralization could improve the reliability of biomedical research.

What Problem Does This Paper Address?

The biomedical research landscape is increasingly concerned about the reliability and reproducibility of published findings. While individual experiments can be reproduced, the cumulative process of building scientific knowledge is often questioned. This paper tackles the problem of evaluating the replicability of a large number of scientific claims at scale, specifically focusing on drug-gene interaction (DGI) claims. The authors aim to understand how the structure of scientific communities, particularly the degree of centralization and collaboration, influences the robustness of these claims. They are addressing a gap in empirical evaluation of how scientific network structures impact the reliability of scientific output.

Methods

The core of this study is a novel strategy to evaluate the replicability of DGI claims by linking published literature with high-throughput experimental data.

  • Claim Extraction: They utilized the Comparative Toxicogenomics Database (CTD) to extract over 239,000 DGI claims from scientific articles, focusing on interactions where a drug either increases or decreases gene expression.
  • High-Throughput Data Integration: These claims were then mapped to the LINCS L1000 dataset, which provides gene expression profiles from experiments using a vast array of chemical perturbagens across different cell lines, dosages, and time points. This allowed for the generation of a "combined z-score" representing the experimental effect size and direction for each DGI.
  • Replication Assessment: A claim was considered "replicated" if the direction of the effect observed in the LINCS L1000 experiment matched the direction reported in the literature.
  • Network Analysis: To assess community structure, they analyzed metadata from the published articles, including author overlap (social independence), methodological overlap (methodological independence), and citation overlap (knowledge independence). Centralization was quantified using the Gini coefficient, measuring the heterogeneity of author contributions to a claim.
  • Statistical Modeling: Logistic regression models were employed to predict replication success based on these network dependency measures, support in the literature, journal prominence, and experimental variability.

Datasets

  • Comparative Toxicogenomics Database (CTD): Used to extract published drug-gene interaction (DGI) claims.
  • LINCS L1000 dataset: A high-throughput experimental dataset measuring gene expression profiles across various conditions, used to generate independent experimental validation of DGI claims.
  • MEDLINE/PubMed and Web of Science databases: Used for extracting author information, methods, and citation data to construct network dependency measures.

The study effectively creates a novel benchmark by linking these large-scale datasets to evaluate claim replicability.

Key Insights and Ideas

  • Centralization Hinders Replication: The most striking insight is that highly centralized scientific communities, characterized by shared authors and similar methodologies, are less likely to produce replicable results. This suggests that a lack of diverse perspectives and approaches can lead to the propagation of less robust findings.
  • Decentralization Fosters Robustness: Conversely, decentralized communities, which involve more independent teams and a wider array of methods, are associated with higher claim replicability. This highlights the value of diverse evidence and independent verification.
  • The "Wisdom of Crowds" in Science: The findings align with the concept of the "wisdom of crowds," where diverse, independent judgments can lead to more accurate collective outcomes. In science, this translates to decentralized communities being more effective at identifying true effects.
  • Confirmation Bias and the "File Drawer Problem": The study provides empirical support for how confirmation bias and the tendency to "file away" negative results can be exacerbated in centralized communities, where a lack of disagreement reinforces existing beliefs.
  • Support in Literature is Necessary but Not Sufficient: While claims with strong support in the literature are more likely to replicate, this effect is significantly amplified in decentralized communities. This means that even widely accepted findings can be unreliable if studied by a monolithic scientific group.

Key Results

  • Single-paper claims replicate 19.0% more frequently than expected.
  • Multi-paper claims replicate 45.5% more frequently than expected.
  • Claims from centralized scientific communities propagate less replicable claims than those from decentralized communities.
  • Support in the literature, social independence, methodological independence, and knowledge independence were all positively associated with claim replication.
  • Centralization was negatively associated with claim replication.
  • In a combined model, centralization and support in the literature largely accounted for the effects of other predictors, underscoring their importance.
  • Claims from decentralized communities showed a 45% higher predicted probability of replication compared to those from centralized communities, even with similar levels of literature support.

Limitations and Future Work

  • Data Limitations: The study acknowledges that the LINCS L1000 data, while valuable, is not entirely free from error. However, they argue its strength lies in its representativeness of DGI research and its freedom from social and methodological dependencies inherent in the publication system.
  • Causality vs. Correlation: While the study identifies strong correlations between community structure and replication, establishing definitive causality is complex. The authors suggest that policies fostering decentralized collaboration are a promising avenue.
  • Generalizability: The study focuses on drug-gene interactions. While the principles might apply more broadly, further research is needed to confirm this across other scientific domains.
  • Future Work: The authors propose extending this approach to a continuous replication system that revises scientific results in light of new experimental evidence, thereby increasing the credibility and robustness of biomedical knowledge. They also suggest exploring how "big science" initiatives, which often involve centralization, can be designed to mitigate potential negative impacts on replicability.

Summary updated

Abstract

Concerns have been expressed about the robustness of experimental findings in several areas of science, but these matters have not been evaluated at scale. Here we identify a large sample of published drug-gene interaction claims curated in the Comparative Toxicogenomics Database (for example, benzo(a)pyrene decreases expression of SLC22A3) and evaluate these claims by connecting them with high-throughput experiments from the LINCS L1000 program. Our sample included 60,159 supporting findings and 4253 opposing findings about 51,292 drug-gene interaction claims in 3363 scientific articles. We show that claims reported in a single paper replicate 19.0% (95% confidence interval [CI], 16.9-21.2%) more frequently than expected, while claims reported in multiple papers replicate 45.5% (95% CI, 21.8-74.2%) more frequently than expected. We also analyze the subsample of interactions with two or more published findings (2493 claims; 6272 supporting findings; 339 opposing findings; 1282 research articles), and show that centralized scientific communities, which use similar methods and involve shared authors who contribute to many articles, propagate less replicable claims than decentralized communities, which use more diverse methods and contain more independent teams. Our findings suggest how policies that foster decentralized collaboration will increase the robustness of scientific findings in biomedical research.
BibTeX
@inproceedings{danchev2019centralized,
  author = {Valentin Danchev and Andrey Rzhetsky and James A. Evans},
  title = {{Centralized scientific communities are less likely to generate replicable results}},
  booktitle = {eLife},
  year = {2019},
  doi = {10.7554/elife.43094},
}

Loading related content...

Figure