Evaluation of compliance of CONSORT-CHM formula 2017 in randomized controlled trials of Chinese herbal medicine formulas: protocol of a five-year review

Juan Wang, Chung Wah Cheng, Yalin Jiao, Dongni Shi, Yaochen Wang, Han Li, Nana Wang, Xihong Wang, Yuqin Li, Liang Feng, Shufeng Luo, Fei Han, Ji Li, Ping Wang, Aiping Lyu, Zhaoxiang Bian, Xuan ZhangFrontiers in Pharmacology 2024 · DOI

This document outlines a study protocol for a systematic review aimed at evaluating how well researchers are following reporting standards for clinical trials involving Chinese herbal medicine.

TL;DR

This paper presents a rigorous protocol for a five-year systematic review to assess compliance with the CONSORT-CHM Formula 2017 reporting guidelines. By analyzing randomized controlled trials (RCTs) published between 2018 and 2022, the authors aim to identify specific gaps in transparency and improve the overall quality of evidence in the field of Traditional Chinese Medicine (TCM).

What Problem Does This Paper Address?

The "gold standard" for medical evidence is the Randomized Controlled Trial (RCT). However, an RCT is only as good as its report; if researchers don't clearly describe their methods, ingredients, or patient selection, the results cannot be validated or replicated.

Chinese Herbal Medicine Formulas (CHMF) present a unique challenge because they are complex mixtures, often tailored to specific "TCM patterns" rather than just Western disease definitions. To address this, the CONSORT-CHM Formula 2017 extension was developed to provide a specialized reporting checklist. Despite its existence, it remains unclear whether the global research community has actually adopted these standards. This paper addresses the need for a systematic, large-scale audit to determine the current state of reporting quality and identify where researchers are still falling short.

Methods

The authors have designed a comprehensive methodology following the Joanna Briggs Institute (JBI) framework and PRISMA-P guidelines:

  1. Systematic Search: A search across eight major English and Chinese databases (including Embase, MEDLINE, CNKI, and Wanfang) for RCTs published from January 2018 to June 2022.
  2. Granular Assessment Tool: Instead of using the 22 broad items from the original CONSORT-CHM checklist, the authors subdivided them into 42 specific sub-questions. This allows for a more objective, binary (yes/no) scoring system.
  3. Rigorous Quality Control:
    • A pilot test on 100 randomly selected articles to refine the process.
  4. Development of a Standard Operating Procedure (SOP) to ensure all reviewers score consistently.
  5. Paired, independent reviews: Two researchers will assess each article "back-to-back," with a third senior researcher resolving any disagreements.
  6. Statistical Analysis: Compliance rates (CR) will be calculated, and subgroup analyses will explore whether factors like publication year, formula type, or country of origin affect reporting quality.

Datasets

As a systematic review protocol, the "dataset" consists of the existing body of published literature. The authors will pull from:

  • International Databases: EBM Reviews, AMED, Embase, Ovid-MEDLINE.
  • Chinese Databases: Wanfang, CNKI, VIP, and CBM.
  • Target Population: Human subjects with any disease or TCM syndrome treated with CHMF (mixtures of two or more medicinal substances).

Key Insights and Ideas

  • The "Black Box" of Ingredients: A major focus of this review is on the "Interventions" section (Items 5a-1 to 5a-11). This includes often-ignored details like the authentication of raw herbs, processing methods (Paozhi), and dosage forms—details critical for pharmacological replication.
  • TCM Pattern Differentiation: The protocol specifically checks if trials reported whether they targeted a specific "TCM Pattern" (e.g., "Spleen Qi Deficiency"), which is a fundamental principle of TCM that is frequently omitted in Western-style reporting.
  • Quantifying Impact: By looking at a five-year window following the release of the 2017 guidelines, the study will effectively measure the "diffusion of innovation" regarding reporting standards in the TCM research community.

Key Results

Note: As this is a study protocol, quantitative results are not yet available.

The expected output will be a comprehensive map of reporting quality across the CHMF domain, identifying which specific CONSORT items (e.g., safety assessment, quality control, or rationale) are most frequently neglected by authors.

Limitations and Future Work

  • Language Bias: The search is limited to English and Chinese. While these cover the majority of TCM research, relevant trials published in Japanese (Kampo) or Korean (TKM) might be missed.
  • Broad Scope: By including all diseases rather than focusing on one (e.g., diabetes), the review may encounter a high volume of heterogeneous data, making it harder to draw disease-specific conclusions.
  • Future Directions: The findings will be used to update the CONSORT-CHM guidelines and develop educational tools for researchers and journal editors to ensure higher transparency in future TCM trials.

Summary updated

Abstract

<b>Background:</b> The CONSORT Extension for Chinese Herbal Medicine Formula 2017 (CONSORT-CHM Formula 2017) has established a reporting standard for randomized controlled trials (RCTs) of Chinese Herbal Medicine Formula (CHMF) interventions; however, its adherence and implications for the design and execution of study design remain ambiguous. It is necessary to evaluate the level of compliance with the CONSORT-CHM Formula 2017 in RCTs conducted over the past 5 years, and to determine the reporting quality of clinical trials in this field. <b>Methods:</b> First, a systematic search is conducted for RCTs on CHMF in EBM Reviews, Allied and Complementary Medicine (AMED), Embase, Ovid-MEDLINE(R), Wanfang data, China National Knowledge Infrastructure (CNKI), VIP Chinese Medical Journal Database (VIP) and Chinese Biomedical Literature (CBM) database, that encompassed CHMF interventional RCTs published from 1 January 2018 to 8 June 2022, with language restriction to English or Chinese. Second, a descriptive analysis will be performed regarding the study design and general characteristics of the included trials. Third, for the quality assessment, we have subdivided the CONSORT-CHM Formula 2017 checklist (consisting of 22 extended items) into a total of 42 sub-questions to facilitate scoring, with a specific focus on the description, quality control, and safety assessment of CHMF interventions. Professional training and a pilot test on 100 randomly selected articles will be provided for all reviewers. Throughout this process, a standard operating procedure (SOP) for quality assessment will be developed to ensure consistency. Each item will be assessed by two reviewers in a paired back-to-back manner, and the compliance rate will be calculated to assess inter-rater agreement. <b>Discussion:</b> This review will identify the current reporting characteristics and quality of CHMF interventional studies and further evaluate the impact of CONSORT-CHM Formula 2017. The results may provide suggestions for future application or promotion of the guideline. Registration: The study has been registered on Open Science Framework (https://osf.io/xpn7f).
BibTeX
@inproceedings{wang2024evaluation,
  author = {Juan Wang and Chung Wah Cheng and Yalin Jiao and Dongni Shi and Yaochen Wang and Han Li and Nana Wang and Xihong Wang and Yuqin Li and Liang Feng and Shufeng Luo and Fei Han and Ji Li and Ping Wang and Aiping Lyu and Zhaoxiang Bian and Xuan Zhang},
  title = {{Evaluation of compliance of CONSORT-CHM formula 2017 in randomized controlled trials of Chinese herbal medicine formulas: protocol of a five-year review}},
  booktitle = {Frontiers in Pharmacology},
  year = {2024},
  doi = {10.3389/fphar.2024.1287262},
}
Figure