Abstract
Large language models (LLMs) have demonstrated remarkable progress in healthcare. However, a significant gap remains regarding LLMs' professionalism in domain-specific clinical practices, limiting their application in real-world diagnostics. In this work, we introduce ZODIAC, an LLM-powered framework with cardiologist-level professionalism designed to engage LLMs in cardiological diagnostics. ZODIAC assists cardiologists by extracting clinically relevant characteristics from patient data, detecting significant arrhythmias, and generating preliminary reports for review and refinement. To achieve cardiologist-level professionalism, ZODIAC is built on a multi-agent collaboration framework, enabling the processing of patient data across multiple modalities. Each LLM agent is fine-tuned using real-world patient data adjudicated by cardiologists, reinforcing the model's professionalism. ZODIAC undergoes rigorous clinical validation with independent cardiologists, evaluated across eight metrics measuring clinical effectiveness and security. Results show that ZODIAC outperforms industry-leading models, including GPT-4o, Llama-3.1-405B, and Gemini-pro, as well as medical-specialist LLMs like BioGPT. ZODIAC demonstrates the transformative potential of specialized LLMs in healthcare by delivering domain-specific solutions that meet the stringent demands of medical practice. Notably, ZODIAC has been successfully integrated into electrocardiography (ECG) devices, exemplifying the trend of embedding LLMs into Software-as-Medical-Device (SaMD).
Introduction: The Professionalism Gap in AI Cardiology
While Large Language Models (LLMs) have shown impressive general medical knowledge, they often fail the "professionalism test" required for real-world clinical practice. In high-stakes environments like cardiology, a "good guess" is insufficient; diagnostics must be precise, concise, and strictly aligned with established medical guidelines. Current general-purpose models struggle with multimodal integration—simultaneously processing tabular clinical metrics and visual ECG tracings—and often fail to adhere to the specific linguistic and structural standards of a cardiologist’s report. Furthermore, meeting the stringent requirements for Software-as-a-Medical-Device (SaMD) as defined by regulatory bodies like the FDA requires a level of reliability and transparency that general models lack. ZODIAC addresses these gaps by moving away from a "one-size-fits-all" model toward a collaborative, multi-agent architecture designed to mimic a professional clinical workflow.
The ZODIAC Framework
The core of ZODIAC is a multi-agent collaboration framework consisting of three specialized LLM agents, each handling a specific part of the diagnostic pipeline.
- Metrics-to-Findings (M2F) Agent: A table-to-text agent (based on Llama-3.1-8B) that extracts characteristics from tabular biostatistical data and metrics such as heart rate and ectopic beat counts.
- Tracings-to-Findings (T2F) Agent: An image-to-text agent (based on LLaVA-v1.5-13B) that analyzes visual ECG waveforms to identify specific arrhythmias.
- Findings-to-Interpretation (F2I) Agent: A text-based agent (Llama-3.1-8B) that synthesizes the outputs of the first two agents, applies clinical guidelines, and generates a final diagnostic interpretation.
This modular design prevents a single model from being "overloaded" by different data modalities, leading to lower diagnostic lability and more consistent results.
Professionalism-Driven Methodology
To ensure cardiologist-level professionalism, ZODIAC incorporates several technical innovations beyond standard model training, focusing on both instruction-tuning and inference-time guidance.
Instruction Fine-Tuning and In-Context Learning
Each agent was fine-tuned on a curated dataset of real-world patient cases adjudicated by human cardiologists. During inference, the system employs In-Context Learning (ICL) using "gold demos"—high-quality examples tailored to the specific patient's demographic (age, gender, arrhythmia class). This ensures the model follows the correct structural templates and uses professional terminology.
Automated Fact-Checking
A critical post-processing step involves an automated fact-checking layer. The system cross-references its findings against a digital library of cardiological guidelines. If a discrepancy is found—for instance, a PR interval duration that does not match the diagnosed "AV block" type—the system triggers a self-correction loop to align the interpretation with medical standards.
Datasets and Expert Adjudication
The researchers eschewed public benchmarks in favor of a proprietary, high-fidelity dataset sourced from collaborating healthcare institutions under IRB-approved protocols. The dataset includes over 2,000 patient cases, balanced across age, gender, and three classes of arrhythmias: Normal, Clinically Significant, and Life-Threatening. Five independent cardiologists reviewed and labeled the data to ensure "professionalism-incorporated" curation, providing a ground truth that reflects actual clinical practice rather than just theoretical knowledge.
Results and Clinical Validation
ZODIAC was evaluated by independent cardiologists against industry leaders (GPT-4o, Gemini-Pro, Llama-3.1-405B) and medical-specialist models (BioGPT, Meditron) across eight axes of professionalism.
- Clinical Superiority: ZODIAC achieved a score of 4.9/5.0 in Comprehensibility and 4.7/5.0 in Organization, significantly higher than GPT-4o (4.2 and 4.1, respectively).
- Reliability: Thanks to the fact-checking layer, ZODIAC scored 4.8/5.0 in being "Free from Hallucination," outperforming all baselines.
- Security and Bias: The model achieved a perfect 5.0/5.0 in "Free from Bias," demonstrating that its training on representative real-world data successfully mitigated common demographic biases.
The results suggest that for specialized medical tasks, targeted fine-tuning and a multi-agent architecture are more effective than raw model size, as ZODIAC (approx. 30B parameters) consistently outperformed the much larger Llama-3.1-405B.
Conclusion and Future Directions
ZODIAC demonstrates the transformative potential of specialized LLMs in healthcare. By delivering domain-specific solutions that meet the stringent demands of medical practice, it bridges the gap between general AI capabilities and clinical utility. While currently specialized for cardiology, the multi-agent blueprint established by ZODIAC offers a path forward for other complex clinical domains like neurology or oncology. Future work will focus on enhancing transparency and exploring defenses against potential adversarial attacks to further solidify trust in AI-assisted diagnostics.
Summary updated