A Multi-agent Large Language Model Framework to Automatically Assess Performance of a Clinical AI Triage Tool

Adam E. Flanders, Yifan Peng, Luciano M. Prevedello, Robyn L. Ball, Errol Colak, Prahlad Menon, George Shih, Hui-Ming Lin, Paras LakhaniarXiv 2025 · arXiv · OpenReview

TL;DR

This paper introduces a novel framework using an ensemble of Large Language Models (LLMs) to automatically assess the performance of clinical AI triage tools. The study demonstrates that this multi-agent approach offers a more reliable and consistent "ground truth" for evaluating AI diagnostic tools compared to relying on a single LLM.

What Problem Does This Paper Address?

Monitoring the performance of clinical AI tools is crucial to detect degradation over time, but establishing a reliable "ground truth" for evaluation is challenging. Manually reviewing vast amounts of medical data is time-consuming and expensive. While LLMs can extract information from clinical reports, their individual performance varies. This paper addresses the need for a robust, automated method to assess AI tool performance, specifically for a commercial intracranial hemorrhage (ICH) detection tool, by leveraging the collective intelligence of multiple LLMs.

Methods

The researchers developed a multi-agent LLM framework. They used eight open-source LLMs and a HIPAA-compliant internal version of GPT-4o. These LLMs were tasked with analyzing the "impression" section of radiology reports to identify the presence and type of intracranial hemorrhage. A "consensus" score was generated based on the agreement among multiple LLMs. Different ensemble configurations (e.g., top-performing models, all models) were tested to determine the most effective for creating a reliable ground truth. Performance was evaluated using metrics like AUC, average precision, F1 score, recall, precision, specificity, and MCC, with human review serving as the ultimate reference standard.

Datasets

The study utilized a dataset of 29,766 non-contrast head CT exams from fourteen hospitals. This dataset was processed by a commercial ICH AI detection tool. A subset of 1,726 exam-report pairs were manually reviewed by human experts to serve as a conservative reference standard for evaluating the LLM performance.

Key Insights and Ideas

  • Ensemble Power: Combining multiple LLMs significantly improves the reliability of assessing AI tool performance compared to using a single LLM. This is akin to having multiple expert reviewers instead of just one.
  • Open-Source Advantage: The study highlights the potential of using readily available open-source LLMs for this task, offering a more accessible and potentially cost-effective solution compared to proprietary models.
  • "Ground Truth" Automation: The LLM ensemble framework provides a promising avenue for automating the creation of a reliable ground truth, which is a major bottleneck in AI performance monitoring.
  • Model Size vs. Performance: While larger models generally performed well, the study noted that model size wasn't the sole determinant of performance, suggesting model architecture and training data also play significant roles.

Key Results

  • The highest Area Under the Curve (AUC) performance was achieved by llama3.3:70b and GPT-4o at 0.78.
  • llama3.3:70b demonstrated the best individual performance with an F1 score of 0.81 and recall of 0.85.
  • The most effective ensemble configurations for assessing performance, measured by MCC, were:
  • Full-9 Ensemble: 0.571
    • Top-3 Ensemble: 0.558
    • Consensus (of eight LLMs): 0.556
  • GPT-4o alone had a lower MCC (0.522) compared to the top ensemble configurations, indicating the benefit of the ensemble approach.
  • No statistically significant differences were found between the Top-3, Full-9, and Consensus ensemble configurations, suggesting flexibility in choosing the ensemble size.

Limitations and Future Work

  • Dataset Imbalance: The dataset had a low prevalence of ICH (4.7%), which can inflate specificity and accuracy values.
  • Incorporation Bias: Radiologists had access to the AI outputs during review, which could introduce bias.
  • Evaluation-Set Bias: Focusing primarily on discordant exams provided a conservative estimate but might not fully represent overall performance.
  • Ambiguous Impressions: Excluding ambiguous report impressions from the analysis introduced some bias.
  • Future Work: The authors suggest further exploration into prompt engineering and the impact of different ensemble sizes and model combinations for optimizing performance. They also highlight the need for continuous monitoring of LLM performance to identify potential failures.

Summary updated

Abstract

Purpose: The purpose of this study was to determine if an ensemble of multiple LLM agents could be used collectively to provide a more reliable assessment of a pixel-based AI triage tool than a single LLM. Methods: 29,766 non-contrast CT head exams from fourteen hospitals were processed by a commercial intracranial hemorrhage (ICH) AI detection tool. Radiology reports were analyzed by an ensemble of eight open-source LLM models and a HIPAA compliant internal version of GPT-4o using a single multi-shot prompt that assessed for presence of ICH. 1,726 examples were manually reviewed. Performance characteristics of the eight open-source models and consensus were compared to GPT-4o. Three ideal consensus LLM ensembles were tested for rating the performance of the triage tool. Results: The cohort consisted of 29,766 head CTs exam-report pairs. The highest AUC performance was achieved with llama3.3:70b and GPT-4o (AUC= 0.78). The average precision was highest for Llama3.3:70b and GPT-4o (AP=0.75 & 0.76). Llama3.3:70b had the highest F1 score (0.81) and recall (0.85), greater precision (0.78), specificity (0.72), and MCC (0.57). Using MCC (95% CI) the ideal combination of LLMs were: Full-9 Ensemble 0.571 (0.552-0.591), Top-3 Ensemble 0.558 (0.537-0.579), Consensus 0.556 (0.539-0.574), and GPT4o 0.522 (0.500-0.543). No statistically significant differences were observed between Top-3, Full-9, and Consensus (p > 0.05). Conclusion: An ensemble of medium to large sized open-source LLMs provides a more consistent and reliable method to derive a ground truth retrospective evaluation of a clinical AI triage tool over a single LLM alone.
BibTeX
@misc{flanders2025multi,
  author = {Adam E. Flanders and Yifan Peng and Luciano M. Prevedello and Robyn L. Ball and Errol Colak and Prahlad Menon and George Shih and Hui-Ming Lin and Paras Lakhani},
  title = {{A Multi-agent Large Language Model Framework to Automatically Assess Performance of a Clinical AI Triage Tool}},
  year = {2025},
  eprint = {2510.26498},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.26498},
}
Figure