Abstract
Chinese medical research has made significant progress regarding statistical analyses over the last decade, but there remains ample room for improvement regarding study designs. This retrospective survey of 2,913 articles from 1998 and 2008 reveals that while statistical accuracy and result presentation have improved, high-quality randomized clinical trials (RCTs) remain rare and often suffer from methodological weaknesses. Urgent implementation of the CONSORT statement is imperative to elevate the standard of clinical evidence.
Introduction
As China’s scientific output exploded in the early 2000s, concerns grew regarding the underlying quality of the research. High-quality clinical evidence requires more than just professional expertise; it demands sound study design and rigorous statistical execution. Without these, even biologically sound work is at risk of producing misleading results. This study sought to quantify the quality gap by surveying leading Chinese medical journals, tracking trends in study design, statistical application, and adherence to international reporting standards.
Methodology
The researchers conducted a comprehensive manual audit of 2,913 original articles from 228 issues of 10 leading Chinese medical journals, all indexed by Medline and sponsored by the Chinese Medical Association. Two cohorts were compared: 1,335 articles from 1998 and 1,578 articles from 2008.
To ensure objectivity, the study employed:
- A Validated Checklist: Developed via the Delphi method to evaluate design, statistics, and interpretation.
- Expert Review: A team of 21 researchers independently reviewed each article in groups of three to resolve discrepancies.
- Standardized Definitions: Criteria from the Oxford Centre for Evidence-Based Medicine were used to categorize study types, while Emerson’s methods categorized statistical tests.
Key Findings
Improvements in Statistical Literacy
There was a marked increase in the use of sophisticated statistics, such as logistic regression and survival analysis. Concurrently, there was a significant decrease in "simple" errors involving t-tests and contingency tables. Statistical accuracy improved from 59.8% in 1998 to 52.2% in 2008 ($\chi^2 = 12.03, p < 0.001$).
The Persistence of Design Defects
Despite a decade of progress, the "gold standard" of randomized clinical trials (RCTs) remains rare, accounting for less than 5% of published work. Furthermore, nearly half of all published studies remained retrospective. While the proportion of studies with design defects fell from 50.9% to 42.4% ($\chi^2 = 21.22, p < 0.001$), the quality of RCT reporting remains a concern. In 2008, over 63% of RCTs failed to report or perform sample size estimations.
Presentation and Interpretation
The most dramatic gain was in result presentation, such as reporting exact p-values rather than binary significance ($p < 0.05$). Inappropriate presentation fell from 92.7% to 78.2%. However, the "trend fallacy"—claiming a "trend of difference" when $p > \alpha$—remains a common error in interpretation that authors are urged to avoid.
Discussion and Limitations
The authors acknowledge a significant selection bias: the review was limited to journals published in Chinese. During this period, many of China's highest-quality researchers began publishing in international English-language journals. Consequently, this survey may represent the "floor" of Chinese research quality rather than its "ceiling."
Conclusion
To bridge the gap between research quantity and quality, the study recommends:
- Mandatory CONSORT Implementation: Journals must enforce international reporting standards for clinical trials.
- Enhanced Education: Strengthening biostatistics and study design training for medical postgraduates.
- Trial Registration: Encouraging the use of public registries to ensure transparency and reduce publication bias.
Summary updated