The LEADING guideline: Reporting standards for expert panel, best-estimate diagnosis, and longitudinal expert all data (LEAD) methods

Veerle Celine Eijsbroek, Katarina Kjell, H. Andrew Schwartz, Jan R. Boehnke, Eiko I. Fried, Daniel N. Klein, Peik Gustafsson, Isabelle Augenstein, Peter Bossuyt, Oscar Kjell

Abstract

In fields such as psychiatry and clinical psychology, the absence of objective biological markers often necessitates the use of "best-estimate" diagnoses derived from expert panels. However, the lack of standardized reporting for these methods—including Expert Panel, Best-Estimate Diagnosis, and Longitudinal Expert All Data (LEAD)—has created a "black box" that hinders reproducibility and validity. This paper introduces the LEADING guideline, a 20-item reporting standard designed to bring transparency to expert-led diagnostic processes.

Introduction

The "gold standard" in medical diagnosis typically relies on objective measures like blood tests or imaging. In contrast, complex conditions in psychiatry and psychology often depend on expert consensus. Researchers utilize various methodologies, such as the Expert Panel, Best-Estimate Diagnosis, and the Longitudinal Expert All Data (LEAD) method, to synthesize diverse data sources into a definitive diagnosis.

Despite their prevalence, these methods suffer from a significant reporting gap. Critical details—such as how experts resolved disagreements or whether they were blinded to specific data—are frequently omitted, undermining the scientific community's ability to evaluate the validity of these diagnostic benchmarks. Without these details, it is impossible to evaluate the validity of the "gold standard" being used to train AI models or validate new diagnostic tools.

Methods

The LEADING (Longitudinal, Evaluation – experts, materials and procedures, and Appropriate Data) guideline was developed through a rigorous four-stage consensus-building process. Initially, a draft was created based on existing literature and input from a patient organization for depression. This was followed by a two-round Delphi procedure involving 27 international experts from diverse academic and methodological backgrounds.

Demographic and professional characteristics of the Delphi expert panel across two rounds.

As shown in Figure 1, the expert pool included specialists in medicine, psychiatry, and artificial intelligence, with significant experience in longitudinal studies and consensus procedures. Following the Delphi rounds, a final consensus meeting was held to refine the language of the standards. To validate the necessity of these guidelines, the authors conducted an audit of 15 randomly selected studies published in 2022 that employed these expert-led methods.

The LEADING Framework

The resulting guideline consists of 20 reporting standards categorized into four primary clusters:

  1. Longitudinal Design (4 items): Detailing the time period, number of data points, and history taken into account.
  2. Appropriate Data (4 items): Describing data triangulation (e.g., combining self-reports with objective tests) and whether experts were blinded to certain measures.
  3. Evaluation (10 items): The core of the guideline, covering expert qualifications, training, conflict of interest, and the specific procedures for solving disagreements (e.g., majority vote vs. consensus).
  4. Validity (2 items): A formal description of the assessment and a reflection on its degree of validity.

Key Findings and Audit Results

The systematic audit revealed a stark lack of transparency in current literature. On average, one-third (33%) of the critical methodological details required by the LEADING guideline were missing from published papers. Specific areas of concern included instructions and training for experts (missing in 73% of cases) and inter-rater reliability (missing in 64% of cases). Furthermore, 87% of audited studies failed to report the "data combination method," essentially the logic used to weigh different pieces of evidence.

Discussion and Limitations

The LEADING guideline serves as a "Rosetta Stone," harmonizing disparate terminology across medicine and psychology. By requiring researchers to move away from the binary "gold standard" label and instead describe validity as a spectrum, the framework encourages more nuanced scientific communication.

However, the development of these guidelines was not without limitations. As illustrated by the participant demographics in Figure 1, the expert panel was predominantly composed of researchers from North America and Europe. This geographic concentration may limit the immediate applicability of the guidelines in different cultural or resource-constrained settings. Future work involves integrating the LEADING guideline into the EQUATOR Network and encouraging journal editors to require the LEADING checklist for submissions.

Conclusion

Standardizing the reporting of expert-led diagnoses is essential for the advancement of clinical research and the development of reliable AI diagnostic tools. The LEADING guideline provides a necessary roadmap for researchers to open the "black box" of consensus-based diagnosis, ensuring that the "best guess" is supported by transparent and reproducible evidence.

Summary updated

Abstract

Accurate assessments of symptoms and illnesses are essential for health research and clinical practice but face many challenges. The absence of a single error-free measure is currently addressed by assessment methods involving experts reviewing several sources of information to achieve a best-estimate assessment. This assessment method is called the Expert Panel method in medicine, and the Best-Estimate Diagnosis or Longitudinal Expert All Data (LEAD) method in psychiatry and psychology. However, due to poor reporting of the assessment methods, the quality of pro-claimed best-estimate assessments is typically difficult to evaluate, and when the method is reported, the reporting quality varies substantially. To tackle this gap, we have developed a reporting guideline following a four-stage approach: 1) drafting reporting standards accompanied by empirical evidence, which were further developed with a patient organization for depression, 2) incorporating expert feedback through a two-round Delphi procedure, 3) refining the guideline based on an expert consensus meeting, and 4) testing the guideline by i) having researchers test it and ii) applying it to previously published studies. The last step also provides evidence for the need for the guideline: 10-63 % (Mean 33 %) of the standards were not reported across thirty randomly selected published studies. The result is the LEADING guideline comprising 20 reporting standards in four groups: the Longitudinal design, the Appropriate data, the Evaluation - experts, materials and procedures, and the Validity group. We hope that the LEADING guideline will assist researchers in planning, conducting, reporting, and evaluating research aiming to achieve best-estimate assessments.
BibTeX
@inproceedings{eijsbroek2025leading,
  author = {Veerle Celine Eijsbroek and Katarina Kjell and H. Andrew Schwartz and Jan R. Boehnke and Eiko I. Fried and Daniel N. Klein and Peik Gustafsson and Isabelle Augenstein and Peter Bossuyt and Oscar Kjell},
  title = {{The LEADING guideline: Reporting standards for expert panel, best-estimate diagnosis, and longitudinal expert all data (LEAD) methods}},
  booktitle = {Comprehensive},
  year = {2025},
  doi = {10.1016/j.comppsych.2025.152603},
}
Figure