Abstract
Clinical communication coding is a vital yet resource-intensive process used to evaluate patient-provider interactions. Traditional methods rely on manual labor that is costly, slow, and prone to inconsistency. This paper introduces MOSAIC (Multi-framOrk Structured Agentic AI system), a multi-agent framework designed to automate this process across diverse clinical contexts. By utilizing specialized agents for planning, retrieval, and verification, MOSAIC achieves an F1 score of 92.8%, matching human-level precision while reducing processing time from over an hour to under five minutes.
Introduction
In clinical research, "communication coding" serves as the gold standard for analyzing the nuances of doctor-patient interactions. By applying specific codebooks, researchers can identify critical behaviors such as empathic responses, shared decision-making, and subtle clinical biases. However, the practical application of these frameworks is hindered by significant logistical barriers. Manual coding is labor-intensive—often requiring three times the duration of the actual conversation—and is susceptible to "coder drift," where interpretations vary between individuals or over time.
While Large Language Models (LLMs) like GPT-4 offer potential for automation, standard single-pass prompting often struggles with the complexity of overlapping frameworks and rare clinical behaviors. MOSAIC addresses these challenges by transitioning from a monolithic prompt to a structured, multi-agent workflow that prioritizes accuracy and consistency.
Methodology: The MOSAIC Architecture
MOSAIC is built on the LangGraph framework, orchestrating four specialized agents to handle the complexities of communication coding:
- Plan Agent: Functions as the central router. It interprets user requirements, selects the appropriate codebooks, and maps the necessary workflow steps.
- Update Agent: Manages the system's knowledge base. When researchers modify a codebook, this agent automatically parses the new rules and updates the underlying vector database.
- Annotation Agent: The primary execution engine. It utilizes Retrieval-Augmented Generation (RAG) to pull specific definitions and "few-shot" examples from a curated library, providing the LLM with the context needed for precise labeling.
- Verification Agent: Acts as a quality gatekeeper. It evaluates labels for consistency and flags low-confidence segments for human intervention.
A central innovation of the system is the Feedback Optimization Loop. This mechanism allows the system to refine its example library based on the Verification Agent’s performance metrics, ensuring the most effective "gold-standard" examples are used for future prompts.
Experimental Design
The system was evaluated using a dataset of 76 patient-provider audio recordings from two distinct clinical domains: Rheumatology, which typically features more structured interactions, and OB/GYN, which often involves higher emotional complexity.
Transcripts were analyzed using five distinct communication frameworks:
- WISER: Focused on clinician empathy and behavior.
- Global: Measuring relational quality, including warmth and respect.
- Intervention (5As): Identifying behavior change strategies (e.g., smoking cessation).
- Patient Behavior: Tracking patient assertiveness and question-asking.
- Bias: Detecting subtle cues of stereotyping or guardedness.
Results and Analysis
MOSAIC demonstrated high performance across all metrics, achieving an overall F1 score of 92.8%. Performance varied slightly by domain, with Rheumatology reaching 96.2% and the more nuanced OB/GYN domain reaching 86.8%.
Key findings include:
- Efficiency Gains: MOSAIC reduced the time required to annotate a 30-minute recording from the human average of 60–90 minutes to just 2–5 minutes.
- Ablation Study: The full agentic system (93.0% F1) significantly outperformed a single-agent RAG baseline (85.9% F1), validating the multi-agent approach.
- Framework Sensitivity: The system was highly effective at identifying "Patient Behavior" (95.1% F1) but found "Global" relational quality (83.1% F1) more challenging, likely due to the abstract nature of those labels.
Discussion and Limitations
A primary limitation identified is the "Voice Gap." Because MOSAIC currently processes text transcripts, it lacks access to prosody—the tone, pitch, and hesitation that often signal warmth or guardedness. Future iterations will aim to integrate multimodal inputs (audio and text) to capture these paralinguistic features.
Additionally, the system must contend with data imbalance, as the vast majority of clinical dialogue does not fall into a specific behavioral category. Maintaining high sensitivity for rare but critical behaviors remains a focus for ongoing development.
Conclusion
MOSAIC represents a significant step toward scalable, high-fidelity clinical communication analysis. By combining agentic orchestration with dynamic few-shot learning, the system provides a robust tool for researchers, allowing human experts to move away from tedious manual labeling and focus on high-level interpretation and intervention.
Summary updated