TL;DR
ZODIAC is a multi-agent large language model (LLM) framework designed to match cardiologist-level expertise in analyzing ECG records and generating clinical diagnostic reports. By decomposing the diagnostic workflow across specialized sub-30B models and grounding outputs in cardiologist-adjudicated data and clinical guidelines, ZODIAC outperforms massive foundation models like GPT-4o and Llama-3.1-405B in clinical accuracy, consistency, and safety.
What Problem Does This Paper Address?
While modern LLMs demonstrate vast medical knowledge, deploying them in life-critical workflows governed by the FDA’s Software-as-a-Medical-Device (SaMD) standards remains challenging. Generic LLMs often produce verbose, disorganized summaries, struggle with multimodal alignment (e.g., combining 24-hour tabular metrics with ECG rhythm strips), and suffer from hallucinations or diagnostic omissions. Meanwhile, existing open-source medical LLMs are predominantly trained on biomedical literature rather than real-world clinical documentation, leaving them ill-suited for daily cardiology practice.
Methods
ZODIAC mirrors a cardiologist's diagnostic thought process through a three-agent multimodal pipeline:
[Tabular Metrics + Biostats] ──> θ_M2F (Llama-3.1-8B) ──┐
├──> Findings (F) ──┐
[ECG Tracing Strips + Biostats] ─> θ_T2F (LLaVA-13B) ──┘ │
├──> θ_F2I (Llama-3.1-8B) + Guidelines (G) ──> Diagnostic Interpretation (I)
│
Fact-Checking Loop
- Metrics-to-Findings Agent ($\theta_{\text{M2F}}$): A fine-tuned Llama-3.1-8B model that extracts natural-language clinical findings from 24-hour summary metrics (e.g., heart rate extremes, ectopic beat counts, atrial fibrillation burden) and patient biostatistics.
- Tracings-to-Findings Agent ($\theta_{\text{T2F}}$): A fine-tuned vision-language model (LLaVA-v1.5-13B) that inspects ECG waveform images (e.g., 10-second rhythm strips) to evaluate intervals (PR, QT), wave morphologies (P-wave, T-wave inversion), and conduction blocks.
- Findings-to-Interpretation Agent ($\theta_{\text{F2I}}$): A fine-tuned Llama-3.1-8B model that synthesizes the itemized findings from both agents and applies evidence-based clinical practice guidelines ($G$) to generate the final diagnostic interpretation.
Key Technical Strategies:
- Instruction Fine-Tuning: Embeds structured, cardiologist-level reasoning directly into the agent weights using peer-reviewed clinical examples.
- Demographic & Clinical In-Context Learning: At inference time, ZODIAC retrieves three exemplar demonstrations matching the specific patient's age group, gender, and arrhythmia category.
- Guideline-Driven Fact-Checking: A backward verification step cross-references generated findings with diagnostic rules (e.g., verifying whether a PR interval $>200\text{ ms}$ properly triggers an interpretation of first-degree AV block). If mismatched, the agents regenerate the offending items.
The complete system is deployed as an AWS-based microservice (API Gateway $\to$ Lambda $\to$ SageMaker) integrated directly into hospital ECG device workflows.
Datasets
The authors avoided public or synthetic benchmarks in favor of a private, real-world clinical dataset:
- Cohort: ECG monitoring records from 2,000+ patients collected under an IRB-approved protocol from partner healthcare institutions.
- Multimodal Inputs: 24-hour tabular metrics alongside representative 10-second ECG strip images.
- Curation & Adjudication: Findings and diagnostic interpretations were drafted by medical research assistants, independently adjudicated by five board-certified cardiologists, and subject to a $\ge 50\%$ peer cross-audit.
- Split: 95% of cases were reserved for instruction tuning, while 5% were held out for blind cardiologist evaluation across three balanced arrhythmia classes:
- Class I (Normal/Benign): e.g., Sinus Bradycardia, Sinus Arrhythmia.
- Class II (Clinically Significant): e.g., PVCs, Pauses ($<3\text{s}$), Atrial Fibrillation.
- Class III (Life-Threatening): e.g., Ventricular Tachycardia, 3rd-Degree AV Block, Ventricular Flutter.
Key Insights and Ideas
- Specialized Multi-Agent beats Brute-Force Scale: With under 30B combined parameters, ZODIAC outperforms generalist models more than ten times its size (Llama-3.1-405B, GPT-4o), as well as a 34B single-agent baseline. Dividing tasks by data modality prevents single-model attention degradation and perceptual overload.
- Literature Pre-training $\neq$ Clinical Practice: Medical foundation models (BioGPT, Meditron-70B, Med42-70B) scored significantly worse than frontier generalist LLMs. Being pre-trained on PubMed abstracts does not teach an LLM the concise, standardized reporting style required for clinical charting.
- Ablation Dynamics: Instruction fine-tuning was found to be the single most critical driver of diagnostic performance, establishing baseline clinical structure. In-context learning provided domain nuance, while guideline-based fact-checking drove hallucination and bias metrics to near-perfect levels.
Key Results
Cardiologists evaluated anonymized model outputs across eight 5-point Likert metrics covering clinical quality and safety:
| Model | Accuracy (ACC) | Completeness (CPL) | Organization (ORG) | Free from Hallucination (FFH) | Free from Bias (FFB) |
|---|---|---|---|---|---|
| GPT-4o | $3.6 \pm 1.0$ | $4.2 \pm 1.0$ | $4.1 \pm 0.7$ | $3.9 \pm 1.0$ | $4.3 \pm 1.0$ |
| Gemini-Pro | $3.7 \pm 1.1$ | $4.1 \pm 1.1$ | $3.9 \pm 1.0$ | $4.3 \pm 1.0$ | $4.2 \pm 1.2$ |
| Llama-3.1-405B | $3.8 \pm 1.2$ | $4.0 \pm 1.0$ | $3.9 \pm 1.0$ | $4.0 \pm 1.0$ | $4.3 \pm 1.0$ |
| Meditron-70B | $3.3 \pm 1.1$ | $3.3 \pm 1.2$ | $3.6 \pm 1.1$ | $3.3 \pm 1.3$ | $3.4 \pm 1.3$ |
| ZODIAC (Ours) | $4.4 \pm 0.4$ | $4.5 \pm 0.7$ | $4.7 \pm 0.4$ | $4.8 \pm 0.3$ | $5.0 \pm 0.0$ |
- Subgroup Equitability: While GPT-4o and Gemini showed performance degradation on life-threatening arrhythmias, geriatric patients, and female cohorts, ZODIAC maintained uniform diagnostic quality across all demographics.
- Diagnostic Stability (Lability): Measuring pairwise sentence-embedding variance across 10 independent generations per case, ZODIAC demonstrated the lowest variance among all tested models, providing predictable and reproducible reports.
Limitations and Future Work
- Assistive Role (MVP): ZODIAC is designed strictly as a clinical assistant for drafting preliminary reports subject to physician sign-off, rather than an autonomous diagnostic system.
- Adversarial & Data Security: The authors note that full compliance with emerging FDA transparency guidelines will require investigating system vulnerabilities against data corruption, prompt injections, and potential backdoor attacks.
- Model Explainability: Future work aims to increase interpretability into the multi-agent decision chain to further establish clinician trust during complex, borderline arrhythmia evaluations.
Summary updated