Zodiac: A Cardiologist-Level LLM Framework for Multi-Agent Diagnostics

Yuan Zhou, Peng Zhang, Mengya Song, Xuwen Zheng, Yiwen Lu, Zhiheng Liu, Yong Chen, Zhaohan XiSubmission to ICLR 2025 · Withdrawn

TL;DR

ZODIAC is a multi-agent large language model (LLM) framework designed to match cardiologist-level expertise in analyzing ECG records and generating clinical diagnostic reports. By decomposing the diagnostic workflow across specialized sub-30B models and grounding outputs in cardiologist-adjudicated data and clinical guidelines, ZODIAC outperforms massive foundation models like GPT-4o and Llama-3.1-405B in clinical accuracy, consistency, and safety.

What Problem Does This Paper Address?

While modern LLMs demonstrate vast medical knowledge, deploying them in life-critical workflows governed by the FDA’s Software-as-a-Medical-Device (SaMD) standards remains challenging. Generic LLMs often produce verbose, disorganized summaries, struggle with multimodal alignment (e.g., combining 24-hour tabular metrics with ECG rhythm strips), and suffer from hallucinations or diagnostic omissions. Meanwhile, existing open-source medical LLMs are predominantly trained on biomedical literature rather than real-world clinical documentation, leaving them ill-suited for daily cardiology practice.

Methods

ZODIAC mirrors a cardiologist's diagnostic thought process through a three-agent multimodal pipeline:

[Tabular Metrics + Biostats] ──> θ_M2F (Llama-3.1-8B) ──┐
                                                         ├──> Findings (F) ──┐
[ECG Tracing Strips + Biostats] ─> θ_T2F (LLaVA-13B)  ──┘                   │
                                                                             ├──> θ_F2I (Llama-3.1-8B) + Guidelines (G) ──> Diagnostic Interpretation (I)
                                                                             │
                                                                      Fact-Checking Loop
  1. Metrics-to-Findings Agent ($\theta_{\text{M2F}}$): A fine-tuned Llama-3.1-8B model that extracts natural-language clinical findings from 24-hour summary metrics (e.g., heart rate extremes, ectopic beat counts, atrial fibrillation burden) and patient biostatistics.
  2. Tracings-to-Findings Agent ($\theta_{\text{T2F}}$): A fine-tuned vision-language model (LLaVA-v1.5-13B) that inspects ECG waveform images (e.g., 10-second rhythm strips) to evaluate intervals (PR, QT), wave morphologies (P-wave, T-wave inversion), and conduction blocks.
  3. Findings-to-Interpretation Agent ($\theta_{\text{F2I}}$): A fine-tuned Llama-3.1-8B model that synthesizes the itemized findings from both agents and applies evidence-based clinical practice guidelines ($G$) to generate the final diagnostic interpretation.

Key Technical Strategies:

  • Instruction Fine-Tuning: Embeds structured, cardiologist-level reasoning directly into the agent weights using peer-reviewed clinical examples.
  • Demographic & Clinical In-Context Learning: At inference time, ZODIAC retrieves three exemplar demonstrations matching the specific patient's age group, gender, and arrhythmia category.
  • Guideline-Driven Fact-Checking: A backward verification step cross-references generated findings with diagnostic rules (e.g., verifying whether a PR interval $>200\text{ ms}$ properly triggers an interpretation of first-degree AV block). If mismatched, the agents regenerate the offending items.

The complete system is deployed as an AWS-based microservice (API Gateway $\to$ Lambda $\to$ SageMaker) integrated directly into hospital ECG device workflows.

Datasets

The authors avoided public or synthetic benchmarks in favor of a private, real-world clinical dataset:

  • Cohort: ECG monitoring records from 2,000+ patients collected under an IRB-approved protocol from partner healthcare institutions.
  • Multimodal Inputs: 24-hour tabular metrics alongside representative 10-second ECG strip images.
  • Curation & Adjudication: Findings and diagnostic interpretations were drafted by medical research assistants, independently adjudicated by five board-certified cardiologists, and subject to a $\ge 50\%$ peer cross-audit.
  • Split: 95% of cases were reserved for instruction tuning, while 5% were held out for blind cardiologist evaluation across three balanced arrhythmia classes:
  • Class I (Normal/Benign): e.g., Sinus Bradycardia, Sinus Arrhythmia.
  • Class II (Clinically Significant): e.g., PVCs, Pauses ($<3\text{s}$), Atrial Fibrillation.
  • Class III (Life-Threatening): e.g., Ventricular Tachycardia, 3rd-Degree AV Block, Ventricular Flutter.

Key Insights and Ideas

  • Specialized Multi-Agent beats Brute-Force Scale: With under 30B combined parameters, ZODIAC outperforms generalist models more than ten times its size (Llama-3.1-405B, GPT-4o), as well as a 34B single-agent baseline. Dividing tasks by data modality prevents single-model attention degradation and perceptual overload.
  • Literature Pre-training $\neq$ Clinical Practice: Medical foundation models (BioGPT, Meditron-70B, Med42-70B) scored significantly worse than frontier generalist LLMs. Being pre-trained on PubMed abstracts does not teach an LLM the concise, standardized reporting style required for clinical charting.
  • Ablation Dynamics: Instruction fine-tuning was found to be the single most critical driver of diagnostic performance, establishing baseline clinical structure. In-context learning provided domain nuance, while guideline-based fact-checking drove hallucination and bias metrics to near-perfect levels.

Key Results

Cardiologists evaluated anonymized model outputs across eight 5-point Likert metrics covering clinical quality and safety:

Model Accuracy (ACC) Completeness (CPL) Organization (ORG) Free from Hallucination (FFH) Free from Bias (FFB)
GPT-4o $3.6 \pm 1.0$ $4.2 \pm 1.0$ $4.1 \pm 0.7$ $3.9 \pm 1.0$ $4.3 \pm 1.0$
Gemini-Pro $3.7 \pm 1.1$ $4.1 \pm 1.1$ $3.9 \pm 1.0$ $4.3 \pm 1.0$ $4.2 \pm 1.2$
Llama-3.1-405B $3.8 \pm 1.2$ $4.0 \pm 1.0$ $3.9 \pm 1.0$ $4.0 \pm 1.0$ $4.3 \pm 1.0$
Meditron-70B $3.3 \pm 1.1$ $3.3 \pm 1.2$ $3.6 \pm 1.1$ $3.3 \pm 1.3$ $3.4 \pm 1.3$
ZODIAC (Ours) $4.4 \pm 0.4$ $4.5 \pm 0.7$ $4.7 \pm 0.4$ $4.8 \pm 0.3$ $5.0 \pm 0.0$
  • Subgroup Equitability: While GPT-4o and Gemini showed performance degradation on life-threatening arrhythmias, geriatric patients, and female cohorts, ZODIAC maintained uniform diagnostic quality across all demographics.
  • Diagnostic Stability (Lability): Measuring pairwise sentence-embedding variance across 10 independent generations per case, ZODIAC demonstrated the lowest variance among all tested models, providing predictable and reproducible reports.

Limitations and Future Work

  • Assistive Role (MVP): ZODIAC is designed strictly as a clinical assistant for drafting preliminary reports subject to physician sign-off, rather than an autonomous diagnostic system.
  • Adversarial & Data Security: The authors note that full compliance with emerging FDA transparency guidelines will require investigating system vulnerabilities against data corruption, prompt injections, and potential backdoor attacks.
  • Model Explainability: Future work aims to increase interpretability into the multi-agent decision chain to further establish clinician trust during complex, borderline arrhythmia evaluations.

Summary updated

Abstract

Large language models (LLMs) have demonstrated remarkable progress in healthcare. However, a significant gap remains regarding LLMs' professionalism in domain-specific clinical practices, limiting their application in real-world diagnostics. In this work, we introduce ZODIAC, an LLM-powered framework with cardiologist-level professionalism designed to engage LLMs in cardiological diagnostics. ZODIAC assists cardiologists by extracting clinically relevant characteristics from patient data, detecting significant arrhythmias, and generating preliminary reports for the review and refinement by cardiologists. To achieve cardiologist-level professionalism, ZODIAC is built on a multi-agent collaboration framework, enabling the processing of patient data across multiple modalities. Each LLM agent is fine-tuned using real-world patient data adjudicated by cardiologists, reinforcing the model's professionalism. ZODIAC undergoes rigorous clinical validation with independent cardiologists, evaluated across eight metrics that measure clinical effectiveness and address security concerns. Results show that ZODIAC outperforms industry-leading models, including OpenAI's GPT-4o, Meta's Llama-3.1-405B, and Google's Gemini-pro, as well as medical-specialist LLMs like Microsoft's BioGPT. ZODIAC demonstrates the transformative potential of specialized LLMs in healthcare by delivering domain-specific solutions that meet the stringent demands of medical practice. Notably, ZODIAC has been successfully integrated into electrocardiography (ECG) devices, exemplifying the growing trend of embedding LLMs into Software-as-Medical-Device (SaMD).
BibTeX
@misc{zhou2024zodiac,
  author = {Yuan Zhou and Peng Zhang and Mengya Song and Xuwen Zheng and Yiwen Lu and Zhiheng Liu and Yong Chen and Zhaohan Xi},
  title = {{Zodiac: A Cardiologist-Level LLM Framework for Multi-Agent Diagnostics}},
  year = {2024},
  eprint = {2410.02026},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.02026},
}

Loading related content...

Figure