The Alberta Risk of Bias Assessment Tool (AQAT:RoB) for the Evaluation of Medical Large Language Model Question-Answer Studies: Development and Pilot Validation (Preprint)

Carrie Ye, J. Ross Mitchell, Daniel C. Baumgart, Zechen Ma, Angela Lim Fung, Daniela Garcia Orellana, Juel Chowdhury, Abass Abdullah, Steven J. Katz, Jacob L. Jaremko, Pierre Boulanger, Claire E.h. Barber, Gillian Lemermeyer, Hosna Jabbari, Lili Mou, Maryam Mirzaei, Mary Waithera Beckett Githumbi, Puneeta Tandon, Randy Goebel, Rhys H. Clark, Whitney Hung, Marjan Abbasi, Farhad Maleki, Scott Klarenbach, Mohamed AbdallaJournal of Medical Internet Research 2025 · DOI

TL;DR

This paper introduces the Alberta Quality Assessment Tool: Risk of Bias (AQAT:RoB), the first tool specifically designed to evaluate the risk of bias in studies using medical large language models (LLMs) for question-answering. The AQAT:RoB was developed through a rigorous process involving literature reviews, expert consensus, and pilot validation, demonstrating promising initial reliability for assessing the validity of LLM-QA studies.

What Problem Does This Paper Address?

The rapid advancement of large language models (LLMs) in healthcare presents immense potential, but their evaluation has lagged behind. While general AI reporting guidelines exist, there's a critical gap in tools specifically designed to assess the risk of bias in studies evaluating LLM question-answer (QA) systems, particularly in medical contexts. Existing risk-of-bias tools, developed for traditional medical research, are not well-suited to the unique challenges posed by LLM-QA studies. This lack of a tailored assessment tool hinders the rigorous evaluation of LLM-QA study validity, potentially leading to premature or harmful adoption of these technologies.

Methods

The development of the AQAT:RoB involved a multi-stage process:

  • Literature Reviews: Two systematic literature reviews were conducted. The first focused on existing quality assessment tools for LLM-QA studies, and the second on LLM-QA studies themselves. These reviews informed the initial draft of the AQAT:RoB.
  • Iterative Refinement: The draft tool underwent a prespecified iterative process including:
  • Modified Delphi Process: An expert panel rated potential bias items on a 5-point scale, providing justifications and suggestions.
  • Consensus Meeting: The steering committee and panelists discussed feedback from the Delphi process, aiming for consensus on inclusion, bias types, and wording.
  • Pilot Validation: The refined tool was piloted on 16 randomly selected studies by independent evaluators.
  • Validation Thresholds: The process aimed to meet an a priori threshold of ≥80% agreement and a Cohen's κ of ≥0.61 between evaluators.

Datasets

The paper does not describe training datasets for LLMs. Instead, it focuses on the development and validation of a tool to assess existing studies. The pilot validation utilized 16 studies that were randomly selected from a pool of 319 studies identified through systematic literature reviews on LLM-QA studies. These studies evaluated LLMs for patient-facing health information.

Key Insights and Ideas

  • The Unique Challenges of LLM-QA Evaluation: Traditional risk-of-bias tools are insufficient because LLM-QA studies have unique methodological and reporting challenges that require a specialized approach.
  • The Importance of "Questions" as a Bias Domain: The AQAT:RoB explicitly includes "Questions" as a domain, recognizing that how questions are formulated, selected, and manipulated can significantly introduce bias, a factor not adequately addressed by existing tools.
  • The Need for Transparency in LLM Answer Generation: The tool emphasizes the importance of understanding how LLM answers are generated and assessed, as evaluating only a subset of answers could lead to misleading conclusions.
  • The Role of Evaluators and Blinding: The AQAT:RoB highlights the critical need for appropriate evaluator selection and blinding to prevent unconscious bias from influencing study outcomes.
  • Industry Influence and Conflicts of Interest: The panel explicitly voted to include "Conflict of Interest" as a consideration, acknowledging the significant influence of industry in AI research and the potential for bias in studies funded or conducted by model vendors.

Key Results

  • AQAT:RoB Structure: The tool consists of 5 high-level domains: Questions, Reference Answers, LLM Answers, Evaluators, and Outcomes. These are further subdivided into 9 subdomains, each with "Support for Judgment" prompts and "Type of Bias" categories.
  • Pilot Validation Success: The pilot validation achieved a percent agreement of 86.1% and a Cohen's κ of 0.70 between evaluators, exceeding the predefined thresholds for promising initial reliability.
  • Domain-Specific Agreement: The "Question Selection" domain showed the highest agreement (93.8%, κ=0.86), while "LLM Answer Selection" had the lowest (68.8%, κ=0.30), indicating areas for potential future refinement.

Limitations and Future Work

  • Geographic Bias in Development: The development team was primarily from Alberta, Canada. Future work should involve a more international group of experts.
  • Medical Focus: The AQAT:RoB was developed with a medical focus and will require validation and adaptation for non-medical LLM-QA studies.
  • Evolving Technology: LLMs are rapidly developing, necessitating periodic updates to the AQAT:RoB to remain relevant.
  • Language Limitations: The tool was developed and evaluated using English-language studies. Translation and validation for other languages are needed.
  • Sample Size for Validation: The pilot validation used a relatively small sample of 16 studies, which may limit the generalizability of the reliability estimates.
  • Future Refinements: The authors suggest future refinements, extensive external validation, and periodic updates to keep pace with evolving technology. They also note that the LLM Answer Selection domain may require further refinement.

Summary updated

BibTeX
@article{ye2025alberta,
  author = {Carrie Ye and J. Ross Mitchell and Daniel C. Baumgart and Zechen Ma and Angela Lim Fung and Daniela Garcia Orellana and Juel Chowdhury and Abass Abdullah and Steven J. Katz and Jacob L. Jaremko and Pierre Boulanger and Claire E.H. Barber and Gillian Lemermeyer and Hosna Jabbari and Lili Mou and Maryam Mirzaei and Mary Waithera Beckett Githumbi and Puneeta Tandon and Randy Goebel and Rhys H. Clark and Whitney Hung and Marjan Abbasi and Farhad Maleki and Scott Klarenbach and Mohamed Abdalla},
  title = {{The Alberta Risk of Bias Assessment Tool (AQAT:RoB) for the Evaluation of Medical Large Language Model Question-Answer Studies: Development and Pilot Validation (Preprint)}},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  doi = {10.2196/87057},
}
Figure