TL;DR
This paper tackles the critical issue of bias in animal research reporting, specifically in stroke models. By developing and validating text-mining tools, the authors found that while reporting of bias-reducing measures has improved in some stroke models (MCAO), it remains stagnant in others (lacunar stroke), highlighting a persistent need for better reporting standards.
What Problem Does This Paper Address?
Reproducibility is a major challenge in biomedical research, with many published findings failing to be replicated. A significant contributor to this problem is the risk of bias in individual studies, which can lead to overstated treatment effects. While guidelines like ARRIVE and checklists exist to improve reporting in animal studies, it's unclear if these efforts are translating into continuous improvements in practice. Specifically, the authors investigate whether studies modeling focal cerebral ischemia (a type of stroke) adequately report measures to reduce bias, such as randomization, blinding, and sample size calculations. Previous work has shown low reporting rates, and this study aims to assess if recent literature shows improvement, particularly by leveraging automated text-mining approaches to make this assessment more efficient.
Methods
The core of this research lies in developing and evaluating text-mining tools, referred to as "regular expressions" (RegEx), to automatically identify the reporting of bias-reducing measures in full-text scientific articles. These RegEx were trained on phrases associated with randomization, blinded assessment of outcome, and sample size calculations. The performance of these tools was then assessed against a "gold standard" manual review of the same articles. Key metrics used for evaluation include sensitivity (how well the tool identifies actual reports), specificity (how well it avoids false positives), and accuracy (overall correctness). The authors also employed standard search strategies on PubMed to identify relevant literature for two specific stroke models: middle cerebral artery occlusion (MCAO) and lacunar stroke.
Datasets
The study utilized two primary datasets of full-text publications:
- MCAO Literature: 918 publications were analyzed for reporting of bias-reducing measures.
- Lacunar Stroke Literature: 46 publications were analyzed.
These datasets were derived from systematic searches on PubMed, with specific date and keyword limitations to capture recent research. The authors also referenced previous systematic reviews and datasets (e.g., from Sena et al. [8], Minnerup et al. [10], and Pedder et al. [12]) for comparative analysis of historical reporting trends.
Key Insights and Ideas
- Uneven Progress: The most striking finding is the divergence in reporting improvements between the two stroke models. While MCAO literature shows positive trends, lacunar stroke literature appears to be stuck in a reporting rut. This suggests that interventions to improve reporting may not be universally effective across different research areas.
- Automated Tools are Promising, but Imperfect: The developed text-mining tools demonstrate good performance, particularly for identifying blinded assessment of outcome. This suggests that automated approaches can significantly streamline the process of assessing reporting quality.
- Sample Size Calculation Remains a Blind Spot: Across both stroke models, the reporting of sample size calculations remains consistently low. This is a critical finding, as adequate sample size is fundamental for study power and reducing the risk of biased results.
- The "Why" Behind Sample Size Reporting: The authors emphasize the importance of sample size calculations not just for statistical power but also for providing confidence that experiments were planned prospectively and not adjusted post-hoc to achieve significance, thus mitigating publication bias.
Key Results
- MCAO Literature:
- Accuracy for automated annotation was 86% for randomization, 94% for blinding, and 100% for sample size calculation.
- Reporting of randomization and sample size calculations showed improvements over time, particularly in the later quartiles of publication dates.
- Lacunar Stroke Literature:
- Accuracy for automated annotation was 67% for randomization, 91% for blinding, and 96% for sample size calculation.
- Reporting of bias-reducing measures in lacunar stroke literature showed no significant improvement since a previous review up to 2012.
- Tool Performance Summary (Table 1):
- MCAO (n=918): Accuracy of 87% for randomization, 90% for blinding, and 95% for sample size calculation.
- Lacunar (n=46): Accuracy of 67% for randomization, 91% for blinding, and 96% for sample size calculation.
Limitations and Future Work
- Single Assessor Bias: The manual "gold standard" ascertainment was performed by a single investigator, introducing a potential for human error.
- Data Accessibility: The study was limited to manuscripts where full-text PDFs were available for automated retrieval, potentially excluding studies with different accessibility or formatting.
- Supplementary Materials: The automated tool did not account for bias reporting in supplementary materials, which might be relevant for some studies.
- Domain-Specific Language: The performance of the RegEx for randomization and sample size calculations was not as strong for stroke studies as in other domains, suggesting potential domain-specific language differences that need to be addressed in future iterations.
- Future Directions: The authors propose integrating their tools with screening platforms that support active learning and topic analysis to further reduce the manual annotation burden. They also suggest crowdsourcing annotations to build a larger dataset for tool development. The ultimate goal is to create near real-time reporting of journal or institutional performance in bias reporting, enabling targeted improvement activities.
Summary updated