Abstract
The reproducibility crisis in biological and medical sciences is a multifaceted problem often rooted in poor experimental reporting. To address this, we developed the Rigor and Transparency Index (RTI), an automated metric derived from SciScore, which uses natural language processing to evaluate manuscripts for compliance with NIH and other reporting guidelines. Analyzing approximately 1.6 million PubMed Central articles from 1997 to 2019, we found that while methodological reporting is improving, it remains generally low. Crucially, the RTI does not correlate with journal Impact Factors, suggesting that prestige does not equate to technical reporting quality.
Introduction
Reliable scientific progress depends on the ability of researchers to replicate previous findings. However, critical details—such as the sex of animal subjects, the use of blinding, or the specific identity of biological reagents—are frequently omitted from published methods sections. While guidelines like ARRIVE and the use of Research Resource Identifiers (RRIDs) have been introduced to improve transparency, monitoring compliance across the vast scientific literature is a daunting task for human reviewers. The Rigor and Transparency Index (RTI) provides a scalable, objective solution to measure methodological quality at the scale of the entire published record.
Methodology: The SciScore Framework
The RTI is based on SciScore, an automated tool that evaluates papers on a 10-point scale. Five points are allocated to Rigor Criteria, including blinding, randomization, power analysis, and reporting sex as a biological variable. The remaining five points are allocated to Key Biological Resources, checking for the unique identifiability of antibodies, cell lines, organisms, and software tools.
The performance of the individual NLP classifiers used in SciScore varies by entity type. As shown in the performance data below, while many classifiers achieve high F1 scores, others—such as power analysis and cell line authentication—are more difficult to detect accurately due to their relative rarity and the complexity of the language used to describe them.
Validation and Reliability
To ensure the tool's accuracy, SciScore's outputs were validated against human-curated data. This comparison demonstrated high levels of overall agreement across most categories, confirming that the automated system can reliably substitute for manual review in large-scale bibliometric studies.
Key Findings: The State of Scientific Reporting
Our analysis of 1.6 million papers reveals a slow but steady upward trend in reporting quality. The average annual SciScore rose from 2.0 ± 0.9 in 1997 to 4.2 ± 1.7 in 2019. Notable improvements were seen in the reporting of randomization (10% to 30%) and the identifiability of software tools (42% to 87%). However, critical rigor criteria like blinding (9%) and power analysis (10%) remain dismally underreported.
The Impact Factor Paradox
One of the most significant findings is the lack of correlation between a journal's RTI and its Journal Impact Factor (JIF). With a Spearman’s rank correlation of -0.11, the data suggests that "high-impact" journals are no more likely to enforce rigorous methodological reporting than their lower-impact counterparts. This highlights a disconnect between a journal's perceived prestige and the technical transparency of its published research.
Conclusion
The RTI offers a new lens through which to view scientific quality, focusing on transparency rather than citations. The data suggests that simply providing guidelines is insufficient to change author behavior; significant improvements in RTI were only observed in journals that actively enforced specific reporting formats. As the scientific community continues to grapple with reproducibility, tools like SciScore and metrics like the RTI will be essential for monitoring progress and incentivizing better reporting practices.
Summary updated