An Agentic AI System for Multi-Framework Communication Coding

Bohao Yang, Rui Yang, Joshua M. Biro, Haoyuan Wang, Jessica L. Handley, Brianna Richardson, Sophia Bessias, Nicoleta J. Economou-Zavlanos, Armando Bedoya, Monica Agrawal, Michael M. Zavlanos, Anand Chowdhury, Raj M. Ratwani, Kai Sun, Kathryn I. Pollak, Michael J. Pencina, Chuan HongarXiv 2025 · DOI · arXiv

Abstract

Clinical communication coding is a vital yet resource-intensive process used to evaluate patient-provider interactions. Traditional methods rely on manual labor that is costly, slow, and prone to inconsistency. This paper introduces MOSAIC (Multi-framOrk Structured Agentic AI system), a multi-agent framework designed to automate this process across diverse clinical contexts. By utilizing specialized agents for planning, retrieval, and verification, MOSAIC achieves an F1 score of 92.8%, matching human-level precision while reducing processing time from over an hour to under five minutes.

Introduction

In clinical research, "communication coding" serves as the gold standard for analyzing the nuances of doctor-patient interactions. By applying specific codebooks, researchers can identify critical behaviors such as empathic responses, shared decision-making, and subtle clinical biases. However, the practical application of these frameworks is hindered by significant logistical barriers. Manual coding is labor-intensive—often requiring three times the duration of the actual conversation—and is susceptible to "coder drift," where interpretations vary between individuals or over time.

While Large Language Models (LLMs) like GPT-4 offer potential for automation, standard single-pass prompting often struggles with the complexity of overlapping frameworks and rare clinical behaviors. MOSAIC addresses these challenges by transitioning from a monolithic prompt to a structured, multi-agent workflow that prioritizes accuracy and consistency.

Methodology: The MOSAIC Architecture

MOSAIC is built on the LangGraph framework, orchestrating four specialized agents to handle the complexities of communication coding:

  1. Plan Agent: Functions as the central router. It interprets user requirements, selects the appropriate codebooks, and maps the necessary workflow steps.
  2. Update Agent: Manages the system's knowledge base. When researchers modify a codebook, this agent automatically parses the new rules and updates the underlying vector database.
  3. Annotation Agent: The primary execution engine. It utilizes Retrieval-Augmented Generation (RAG) to pull specific definitions and "few-shot" examples from a curated library, providing the LLM with the context needed for precise labeling.
  4. Verification Agent: Acts as a quality gatekeeper. It evaluates labels for consistency and flags low-confidence segments for human intervention.

A central innovation of the system is the Feedback Optimization Loop. This mechanism allows the system to refine its example library based on the Verification Agent’s performance metrics, ensuring the most effective "gold-standard" examples are used for future prompts.

Experimental Design

The system was evaluated using a dataset of 76 patient-provider audio recordings from two distinct clinical domains: Rheumatology, which typically features more structured interactions, and OB/GYN, which often involves higher emotional complexity.

Transcripts were analyzed using five distinct communication frameworks:

  • WISER: Focused on clinician empathy and behavior.
  • Global: Measuring relational quality, including warmth and respect.
  • Intervention (5As): Identifying behavior change strategies (e.g., smoking cessation).
  • Patient Behavior: Tracking patient assertiveness and question-asking.
  • Bias: Detecting subtle cues of stereotyping or guardedness.

Results and Analysis

MOSAIC demonstrated high performance across all metrics, achieving an overall F1 score of 92.8%. Performance varied slightly by domain, with Rheumatology reaching 96.2% and the more nuanced OB/GYN domain reaching 86.8%.

Key findings include:

  • Efficiency Gains: MOSAIC reduced the time required to annotate a 30-minute recording from the human average of 60–90 minutes to just 2–5 minutes.
  • Ablation Study: The full agentic system (93.0% F1) significantly outperformed a single-agent RAG baseline (85.9% F1), validating the multi-agent approach.
  • Framework Sensitivity: The system was highly effective at identifying "Patient Behavior" (95.1% F1) but found "Global" relational quality (83.1% F1) more challenging, likely due to the abstract nature of those labels.

Discussion and Limitations

A primary limitation identified is the "Voice Gap." Because MOSAIC currently processes text transcripts, it lacks access to prosody—the tone, pitch, and hesitation that often signal warmth or guardedness. Future iterations will aim to integrate multimodal inputs (audio and text) to capture these paralinguistic features.

Additionally, the system must contend with data imbalance, as the vast majority of clinical dialogue does not fall into a specific behavioral category. Maintaining high sensitivity for rare but critical behaviors remains a focus for ongoing development.

Conclusion

MOSAIC represents a significant step toward scalable, high-fidelity clinical communication analysis. By combining agentic orchestration with dynamic few-shot learning, the system provides a robust tool for researchers, allowing human experts to move away from tedious manual labeling and focus on high-level interpretation and intervention.

Summary updated

Abstract

Clinical communication is central to patient outcomes, yet large-scale human annotation of patient-provider conversation remains labor-intensive, inconsistent, and difficult to scale. Existing approaches based on large language models typically rely on single-task models that lack adaptability, interpretability, and reliability, especially when applied across various communication frameworks and clinical domains. In this study, we developed a Multi-framework Structured Agentic AI system for Clinical Communication (MOSAIC), built on a LangGraph-based architecture that orchestrates four core agents, including a Plan Agent for codebook selection and workflow planning, an Update Agent for maintaining up-to-date retrieval databases, a set of Annotation Agents that applies codebook-guided retrieval-augmented generation (RAG) with dynamic few-shot prompting, and a Verification Agent that provides consistency checks and feedback. To evaluate performance, we compared MOSAIC outputs against gold-standard annotations created by trained human coders. We developed and evaluated MOSAIC using 26 gold standard annotated transcripts for training and 50 transcripts for testing, spanning rheumatology and OB/GYN domains. On the test set, MOSAIC achieved an overall F1 score of 0.928. Performance was highest in the Rheumatology subset (F1 = 0.962) and strongest for Patient Behavior (e.g., patients asking questions, expressing preferences, or showing assertiveness). Ablations revealed that MOSAIC outperforms baseline benchmarking.
BibTeX
@misc{yang2025agentic,
  author = {Bohao Yang and Rui Yang and Joshua M. Biro and Haoyuan Wang and Jessica L. Handley and Brianna Richardson and Sophia Bessias and Nicoleta J. Economou-Zavlanos and Armando Bedoya and Monica Agrawal and Michael M. Zavlanos and Anand Chowdhury and Raj M. Ratwani and Kai Sun and Kathryn I. Pollak and Michael J. Pencina and Chuan Hong},
  title = {{An Agentic AI System for Multi-Framework Communication Coding}},
  year = {2025},
  eprint = {2512.08659},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.08659},
}
Figure