Abstract
In fields such as psychiatry and clinical psychology, the absence of objective biological markers often necessitates the use of "best-estimate" diagnoses derived from expert panels. However, the lack of standardized reporting for these methods—including Expert Panel, Best-Estimate Diagnosis, and Longitudinal Expert All Data (LEAD)—has created a "black box" that hinders reproducibility and validity. This paper introduces the LEADING guideline, a 20-item reporting standard designed to bring transparency to expert-led diagnostic processes.
Introduction
The "gold standard" in medical diagnosis typically relies on objective measures like blood tests or imaging. In contrast, complex conditions in psychiatry and psychology often depend on expert consensus. Researchers utilize various methodologies, such as the Expert Panel, Best-Estimate Diagnosis, and the Longitudinal Expert All Data (LEAD) method, to synthesize diverse data sources into a definitive diagnosis.
Despite their prevalence, these methods suffer from a significant reporting gap. Critical details—such as how experts resolved disagreements or whether they were blinded to specific data—are frequently omitted, undermining the scientific community's ability to evaluate the validity of these diagnostic benchmarks. Without these details, it is impossible to evaluate the validity of the "gold standard" being used to train AI models or validate new diagnostic tools.
Methods
The LEADING (Longitudinal, Evaluation – experts, materials and procedures, and Appropriate Data) guideline was developed through a rigorous four-stage consensus-building process. Initially, a draft was created based on existing literature and input from a patient organization for depression. This was followed by a two-round Delphi procedure involving 27 international experts from diverse academic and methodological backgrounds.
As shown in Figure 1, the expert pool included specialists in medicine, psychiatry, and artificial intelligence, with significant experience in longitudinal studies and consensus procedures. Following the Delphi rounds, a final consensus meeting was held to refine the language of the standards. To validate the necessity of these guidelines, the authors conducted an audit of 15 randomly selected studies published in 2022 that employed these expert-led methods.
The LEADING Framework
The resulting guideline consists of 20 reporting standards categorized into four primary clusters:
- Longitudinal Design (4 items): Detailing the time period, number of data points, and history taken into account.
- Appropriate Data (4 items): Describing data triangulation (e.g., combining self-reports with objective tests) and whether experts were blinded to certain measures.
- Evaluation (10 items): The core of the guideline, covering expert qualifications, training, conflict of interest, and the specific procedures for solving disagreements (e.g., majority vote vs. consensus).
- Validity (2 items): A formal description of the assessment and a reflection on its degree of validity.
Key Findings and Audit Results
The systematic audit revealed a stark lack of transparency in current literature. On average, one-third (33%) of the critical methodological details required by the LEADING guideline were missing from published papers. Specific areas of concern included instructions and training for experts (missing in 73% of cases) and inter-rater reliability (missing in 64% of cases). Furthermore, 87% of audited studies failed to report the "data combination method," essentially the logic used to weigh different pieces of evidence.
Discussion and Limitations
The LEADING guideline serves as a "Rosetta Stone," harmonizing disparate terminology across medicine and psychology. By requiring researchers to move away from the binary "gold standard" label and instead describe validity as a spectrum, the framework encourages more nuanced scientific communication.
However, the development of these guidelines was not without limitations. As illustrated by the participant demographics in Figure 1, the expert panel was predominantly composed of researchers from North America and Europe. This geographic concentration may limit the immediate applicability of the guidelines in different cultural or resource-constrained settings. Future work involves integrating the LEADING guideline into the EQUATOR Network and encouraging journal editors to require the LEADING checklist for submissions.
Conclusion
Standardizing the reporting of expert-led diagnoses is essential for the advancement of clinical research and the development of reliable AI diagnostic tools. The LEADING guideline provides a necessary roadmap for researchers to open the "black box" of consensus-based diagnosis, ensuring that the "best guess" is supported by transparent and reproducible evidence.
Summary updated