Bias in the reporting of sex and age in biomedical research on mouse models

Oscar Flórez-Vargas, Andy Brass, George Karystianis, Michael Bramhall, Robert Stevens, Sheena Cruickshank, Goran NenadićeLife 2016 · DOI

TL;DR

This study uses text mining to analyze over 15,000 biomedical research papers on mouse models, revealing that while reporting of animal sex and age has increased, only about half of papers in 2014 included this crucial information. The research highlights significant sex-bias in reporting across different disease areas, with cardiovascular disease models favoring males and infectious disease models favoring females, underscoring the need for improved methodological transparency to enhance research reproducibility.

What Problem Does This Paper Address?

Translating findings from animal models to human treatments is a major challenge in biomedical research, with a high failure rate in clinical trials. A key contributor to this translational gap is the lack of reproducibility in preclinical studies. The authors argue that inadequate reporting of experimental methods and materials, specifically the sex and age of the animals used, is a significant factor hindering reproducibility. These biological variables are known to influence disease phenotypes, susceptibility, and treatment responses, making their accurate reporting essential for other researchers to build upon existing work and for understanding sex-specific differences in disease.

Methods

The core of this study is a large-scale text mining approach to automatically extract information about the sex and age of mice from research papers. The researchers developed a rule-based system using GATE (General Architecture for Text Engineering) to identify mentions of sex and age based on lexical patterns. This system was trained and validated on a subset of articles. To assess the system's performance, they calculated precision, recall, and F-scores, achieving high accuracy (around 92%) for both sex and age. The text mining system was then applied to a corpus of over 15,000 full-text articles published between 1994 and 2014.

Datasets

The primary dataset consists of 15,311 full-text articles from the PubMed Central Open Access subset, published between 1994 and 2014, focusing on mouse experimentation. This corpus was selected to ensure access to full text for analysis. The researchers also created several smaller datasets for specific purposes:

  • Training and development sets: Used for designing and refining the text mining rules (Supplementary file 2A, 2C).
  • Manual inspection set: Used to validate the accuracy of the text mining system's output (Supplementary file 2B).
  • Evaluation dataset: A set of 50 articles used to formally evaluate the performance of the text mining system (Supplementary file 2D).
  • Disease-specific datasets: Subsets of the main corpus were analyzed for six specific disease groups (cardiovascular diseases, cancer, diabetes mellitus, lung diseases, infectious diseases, and neurological disorders) and further broken down by research approach (genetics, immunology, physiopathology, and therapy).

Key Insights and Ideas

  • The "Sex and Age" Gap is Real and Persistent: Despite increasing awareness, a significant portion of research papers still fail to report basic information about the sex and age of the mice used, hindering reproducibility.
  • Text Mining as a Scalable Solution: This study demonstrates the power of text mining to systematically analyze large volumes of literature, providing quantitative insights into reporting practices that manual review would find infeasible.
  • Sex Bias Varies by Disease: The study reveals that the bias in reporting sex is not uniform across all research areas. This suggests that the underlying biological or experimental reasons for focusing on one sex over another differ depending on the disease.
  • Guidelines Have an Impact, But Aren't Enough: While reporting has improved, the introduction of guidelines like ARRIVE hasn't completely solved the problem, indicating a need for continued efforts and potentially stricter enforcement.

Key Results

  • Reporting Trends: The frequency of papers reporting both sex and age has increased over two decades, but only reached approximately 50% by 2014.
  • Overall Sex Reporting: Historically, female mice have been reported more often than male mice, reaching a plateau of about 33% in the last decade.
  • Disease-Specific Reporting:
    • Studies on diabetes showed the highest frequency of sex and age reporting (68%), while cancer studies had the lowest (48%).
  • Neurological disorder studies generally showed the best reporting.
  • Sex Bias in Disease Models:
    • Male-bias: Strongest in cardiovascular disease models (2.25:1 ratio of male to female reporting).
  • Female-bias: Strongest in infectious disease models (3.54:1 ratio of female to male reporting).
  • Research Approach Bias: There's a preference for using female models in immunology research and male models in genetics research.
  • No Correlation with Journal Metrics: The study found no correlation between the quality of sex/age reporting and journal impact factor or h-index, suggesting that journal prestige doesn't guarantee methodological transparency.
  • Text Mining Performance: The text mining system achieved an F-score of approximately 92% for both sex and age identification.

Limitations and Future Work

  • Dataset Limitations: The study relied on the PubMed Central Open Access subset, which may not represent the entire biomedical literature. Some journals deposit only a portion of their content in this subset.
  • Text Mining Nuances: While effective, the text mining system had some false positives and negatives, particularly with complex age expressions or inferred sex mentions (e.g., "pregnant"). The authors suggest refining the rules for better capture.
  • Future Directions: The authors propose that their text mining strategy can be extended to explore other aspects of reporting and potentially used for pre-screening manuscripts submitted to journals. They also suggest that the insights gained can help identify specific areas where reporting needs improvement, contributing to more cost-effective efforts to enhance reproducibility.

Summary updated

Abstract

In animal-based biomedical research, both the sex and the age of the animals studied affect disease phenotypes by modifying their susceptibility, presentation and response to treatment. The accurate reporting of experimental methods and materials, including the sex and age of animals, is essential so that other researchers can build on the results of such studies. Here we use text mining to study 15,311 research papers in which mice were the focus of the study. We find that the percentage of papers reporting the sex and age of mice has increased over the past two decades: however, only about 50% of the papers published in 2014 reported these two variables. We also compared the quality of reporting in six preclinical research areas and found evidence for different levels of sex-bias in these areas: the strongest male-bias was observed in cardiovascular disease models and the strongest female-bias was found in infectious disease models. These results demonstrate the ability of text mining to contribute to the ongoing debate about the reproducibility of research, and confirm the need to continue efforts to improve the reporting of experimental methods and materials.
BibTeX
@inproceedings{florezvargas2016bias,
  author = {Oscar Flórez-Vargas and Andy Brass and George Karystianis and Michael Bramhall and Robert Stevens and Sheena Cruickshank and Goran Nenadić},
  title = {{Bias in the reporting of sex and age in biomedical research on mouse models}},
  booktitle = {eLife},
  year = {2016},
  doi = {10.7554/elife.13615},
}
Figure