Centralized "big science" communities more likely generate non-replicable results

Valentin Danchev, Andrey Rzhetsky, James A. EvansarXiv 2018 · arXiv

Abstract

The "reproducibility crisis" in biomedicine and psychology has often been attributed to individual failures like p-hacking or low statistical power. However, this study investigates the systemic impact of scientific community social structures. By evaluating over 50,000 published drug-gene interaction claims against high-throughput robotic experiments (NIH LINCS L1000), researchers found that centralization—where a few "star" scientists and shared methods dominate a field—significantly hinders replicability. Robust scientific truth is more effectively established by decentralized communities of independent teams utilizing diverse methodologies.

Introduction: The Social Roots of the Reproducibility Crisis

A significant portion of published results in biomedicine cannot be replicated. While previous analyses focused on statistical power or researcher bias, this paper explores whether the trend toward "Big Science"—massive collaborations and shared infrastructure—is itself a risk factor. The authors hypothesize that highly connected communities can act as echo chambers, where social dependencies and shared methodological biases propagate "fragile" findings as a false consensus.

Methodology: A High-Throughput Replication Framework

To test this hypothesis, researchers developed an automated verification pipeline that avoids the slow and expensive process of manual lab replication.

Claim Extraction and Support Distribution

The team used the Comparative Toxicogenomics Database (CTD) to extract over 51,000 claims regarding drug-gene interactions. They analyzed the literature to determine the level of support or opposition for each claim, building a multilayer network to assess social and methodological dependencies between papers.

Figure A shows the distribution of claims by published findings; Figure B illustrates the probability of support in the literature for specific drug-gene claims.

Automated Verification

The NIH LINCS L1000 program, containing 1.3 million gene expression profiles generated by standardized robotic systems, served as the "ground truth." Published claims were compared against this benchmark to determine their replication rate (RR).

Key Results: The Centralization Penalty

The study revealed a stark contrast between literature consensus and experimental reality.

Replication Rates and Literature Support

Claims with "Very High" support in the literature were found to replicate 45% more frequently than random chance. Conversely, "Contradictory" claims—where papers in the literature disagreed—were significantly less likely to replicate than random benchmarks.

Figure D displays the distribution of effect sizes in the LINCS L1000 experiment; Figure E shows the replication rates (RR) across different levels of literature support compared to random chance.

The Impact of Community Structure

  • Independence Advantage: Social independence (OR 6.31) and methodological independence (OR 6.30) were the strongest predictors of whether a claim would replicate.
  • Centralization Penalty: As community centralization increases (measured by the Gini coefficient of author dominance), the probability of replication drops sharply (OR 0.36).
  • The "Aha" Moment: A claim with wide support from a centralized community is statistically indistinguishable in its replication probability from a claim reported in a single, isolated paper.

Discussion: Collective Correction vs. Echo Chambers

Science relies on a "collective correction" mechanism. In decentralized communities, different labs act as independent checks on each other. Centralization breaks this mechanism; overlapping collaborations and shared protocols create an environment where systematic biases are repeated rather than corrected. Consensus in centralized fields often reflects social proximity rather than objective truth.

Conclusion and Policy Implications

The findings suggest that science policy should prioritize funding for decentralized competition and intellectual diversity. While large-scale infrastructure is necessary for modern research, maintaining social and methodological independence is crucial for ensuring that "Big Science" produces robust results. Future research should explore these dynamics in other fields, such as high-energy physics, where centralization is structurally unavoidable.

Summary updated

BibTeX
@misc{danchev2018centralized,
  author = {Valentin Danchev and Andrey Rzhetsky and James A. Evans},
  title = {{Centralized "big science" communities more likely generate non-replicable results}},
  year = {2018},
  eprint = {1801.05042},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1801.05042},
}
Figure