Rigor and Transparency Index, a new metric of quality for assessing biological and medical science methods

Joseph Menke, Martijn Roelandse, İbrahim Burak Özyurt, Maryann E. Martone, Anita BandrowskibioRxiv 2020 · DOI

Abstract

The reproducibility crisis in biological and medical sciences is a multifaceted problem often rooted in poor experimental reporting. To address this, we developed the Rigor and Transparency Index (RTI), an automated metric derived from SciScore, which uses natural language processing to evaluate manuscripts for compliance with NIH and other reporting guidelines. Analyzing approximately 1.6 million PubMed Central articles from 1997 to 2019, we found that while methodological reporting is improving, it remains generally low. Crucially, the RTI does not correlate with journal Impact Factors, suggesting that prestige does not equate to technical reporting quality.

Introduction

Reliable scientific progress depends on the ability of researchers to replicate previous findings. However, critical details—such as the sex of animal subjects, the use of blinding, or the specific identity of biological reagents—are frequently omitted from published methods sections. While guidelines like ARRIVE and the use of Research Resource Identifiers (RRIDs) have been introduced to improve transparency, monitoring compliance across the vast scientific literature is a daunting task for human reviewers. The Rigor and Transparency Index (RTI) provides a scalable, objective solution to measure methodological quality at the scale of the entire published record.

Methodology: The SciScore Framework

The RTI is based on SciScore, an automated tool that evaluates papers on a 10-point scale. Five points are allocated to Rigor Criteria, including blinding, randomization, power analysis, and reporting sex as a biological variable. The remaining five points are allocated to Key Biological Resources, checking for the unique identifiability of antibodies, cell lines, organisms, and software tools.

The performance of the individual NLP classifiers used in SciScore varies by entity type. As shown in the performance data below, while many classifiers achieve high F1 scores, others—such as power analysis and cell line authentication—are more difficult to detect accurately due to their relative rarity and the complexity of the language used to describe them.

Performance metrics (F1, Precision, Recall) for the various automated classifiers used to calculate the Rigor and Transparency Index.

Validation and Reliability

To ensure the tool's accuracy, SciScore's outputs were validated against human-curated data. This comparison demonstrated high levels of overall agreement across most categories, confirming that the automated system can reliably substitute for manual review in large-scale bibliometric studies.

Validation of SciScore results against human curators, showing high agreement rates across rigor and resource categories.

Key Findings: The State of Scientific Reporting

Our analysis of 1.6 million papers reveals a slow but steady upward trend in reporting quality. The average annual SciScore rose from 2.0 ± 0.9 in 1997 to 4.2 ± 1.7 in 2019. Notable improvements were seen in the reporting of randomization (10% to 30%) and the identifiability of software tools (42% to 87%). However, critical rigor criteria like blinding (9%) and power analysis (10%) remain dismally underreported.

The Impact Factor Paradox

One of the most significant findings is the lack of correlation between a journal's RTI and its Journal Impact Factor (JIF). With a Spearman’s rank correlation of -0.11, the data suggests that "high-impact" journals are no more likely to enforce rigorous methodological reporting than their lower-impact counterparts. This highlights a disconnect between a journal's perceived prestige and the technical transparency of its published research.

Conclusion

The RTI offers a new lens through which to view scientific quality, focusing on transparency rather than citations. The data suggests that simply providing guidelines is insufficient to change author behavior; significant improvements in RTI were only observed in journals that actively enforced specific reporting formats. As the scientific community continues to grapple with reproducibility, tools like SciScore and metrics like the RTI will be essential for monitoring progress and incentivizing better reporting practices.

Summary updated

Abstract

Abstract The reproducibility crisis in science is a multifaceted problem involving practices and incentives, both in the laboratory and in publication. Fortunately, some of the root causes are known and can be addressed by scientists and authors alike. After careful consideration of the available literature, the National Institutes of Health identified several key problems with the way that scientists conduct and report their research and introduced guidelines to improve the rigor and reproducibility of pre-clinical studies. Many journals have implemented policies addressing these same criteria. We currently have, however, no comprehensive data on how these guidelines are impacting the reporting of research. Using SciScore, an automated tool developed to review the methods sections of manuscripts for the presence of criteria associated with the NIH and other reporting guidelines, e.g., ARRIVE, RRIDs, we have analyzed ~1.6 million PubMed Central papers to determine the degree to which articles were addressing these criteria. The tool scores each paper on a ten point scale identifying sentences that are associated with compliance with criteria associated with increased rigor (5 pts) and those associated with key resource identification and authentication (5 pts). From these data, we have built the Rigor and Transparency Index, which is the average score for analyzed papers in a particular journal. Our analyses show that the average score over all journals has increased since 1997, but remains below five, indicating that less than half of the rigor and reproducibility criteria are routinely addressed by authors. To analyze the data further, we examined the prevalence of individual criteria across the literature, e.g., the reporting of a subject’s sex (21-37% of studies between 1997 and 2019), the inclusion of sample size calculations (2-10%), whether the study addressed blinding (3-9%), or the identifiability of key biological resources such as antibodies (11-43%), transgenic organisms (14-22%), and cell lines (33-39%). The greatest increase in prevalence for rigor criteria was seen in the use of randomization of subjects (10-30%), while software tool identifiability improved the most among key resource types (42-87%). We further analyzed individual journals over time that had implemented specific author guidelines covering rigor criteria, and found that in some journals, they had a big impact, whereas in others they did not. We speculate that unless they are enforced, author guidelines alone do little to improve the number of criteria addressed by authors. Our Rigor and Transparency Index did not correlate with the impact factors of journals.
BibTeX
@misc{menke2020rigor,
  author = {Joseph Menke and Martijn Roelandse and İbrahim Burak Özyurt and Maryann E. Martone and Anita Bandrowski},
  title = {{Rigor and Transparency Index, a new metric of quality for assessing biological and medical science methods}},
  year = {2020},
  doi = {10.1101/2020.01.15.908111},
  howpublished = {bioRxiv},
}
Figure