TL;DR
This paper introduces CAFA, a novel system that uses a multi-agent Large Language Model (LLM) workflow to provide personalized, real-time hearing aid adjustments. By combining live ambient audio analysis, audiograms, and user feedback in a conversational manner, CAFA aims to overcome the limitations of static hearing aid fittings in dynamic acoustic environments.
What Problem Does This Paper Address?
Modern hearing aids, while advanced, are typically fitted once in a clinic and then operate with static programs. This approach struggles to adapt to the ever-changing acoustic environments people encounter daily, leading to suboptimal hearing experiences. While some hearing aids attempt automatic program switching based on ambient sound, misclassifications can degrade perception. Human-in-the-loop remote adjustments are costly and not scalable. This paper addresses the need for an automated, context-aware system that can dynamically adjust hearing aid settings in real-time, ensuring both effectiveness and safety.
Methods
CAFA employs a sophisticated multi-agent LLM workflow for context-adaptive hearing aid fitting. The core components include:
- Ambient Sound Recognition: A lightweight neural network, fine-tuned on YAMNet embeddings, classifies live audio streams into "conversation," "noise," or "quiet" with high accuracy. This real-time classification provides crucial environmental context.
- Multi-Agent LLM Workflow:
- Context Acquisition: Fuses the user's audiogram with the detected ambient sound context into a global state vector.
- Subproblem Classifier: Categorizes user complaints into one of six predefined fitting challenges (e.g., noise, distortion, clarity) using an LLM guided by a structured prompt. This ensures structured input for subsequent steps.
- Strategy Provider: Translates the classified subproblem into a concrete, step-by-step counseling and fitting plan. It uses a "strategy book" (knowledge base) and an LLM-driven slot-filling dialogue manager to gather necessary information from the user through natural conversation. It also incorporates a conflict resolution mechanism for inconsistent user input.
- Ethical Regulator: Reviews the proposed script and JSON output for clinical safety, template compliance, and user-centered language before presenting it to the user.
- LLM as a Judge: An independent LLM evaluates the entire output based on predefined metrics for clinical correctness, usability, and technical validity, ensuring quality assurance.
Datasets
- Ambient Sound Classification: The system was trained on a three-class dataset comprising conversation, noise, and quiet audio samples. Conversational and noise clips were sourced from the Microsoft Scalable Noisy Speech Dataset (MS-SNSD), while quiet samples were self-recorded.
- Synthetic Simulated Fitting Workflow: For initial testing, 130 synthetic sessions were created. Audiometric profiles were sampled from the ISO 7029 distribution, combined with six canonical problem types. Ground-truth fitting actions were determined by two licensed audiologists.
Key Insights and Ideas
- Ambient Sound as a Critical Context: The paper strongly emphasizes that real-time awareness of the acoustic environment is paramount for effective hearing aid fitting, moving beyond static profiles.
- LLMs for Nuanced Dialogue and Reasoning: LLMs are leveraged not just for generating text but for their ability to perform multi-step reasoning, understand nuanced user complaints, and engage in adaptive dialogue to gather precise information.
- Modular Multi-Agent System for Safety and Efficiency: The modular design with distinct agents for classification, strategy provision, and ethical regulation, overseen by an LLM Judge, ensures that adjustments are both personalized and clinically safe.
- Bridging the Gap Between User Experience and Clinical Rigor: CAFA aims to provide a user-friendly, conversational experience while maintaining the precision and safety expected from clinical audiologists.
Key Results
- Ambient Sound Classification Accuracy: The lightweight classifier achieved an overall accuracy of 91.2%, with macro-averaged precision, recall, and F1 scores of 0.914, 0.903, and 0.908, respectively.
- LLM Judge Performance: The system achieved impressive average scores from the LLM Judge: 0.73 for Template Compliance (STC), 4.25 for Clinical Safety (SCS), 3.07 for Personalization Adequacy (SPA), 4.76 for Readability & Empathy (SRE), and 0.95 for Internal Consistency (SIC).
- Ablation Study: Removing the ambient sound parser agent significantly increased dialogue length, highlighting its crucial role in conversational efficiency.
Limitations and Future Work
The authors acknowledge several limitations:
- Dataset Modesty and English-Centricity: The sound dataset is relatively small and primarily English-focused, which may impact performance in non-Western acoustic scenes.
- Synthetic Users vs. Real Human Behavior: Synthetic users in the simulation cannot fully replicate the complexities of real human interaction and feedback.
Future work will focus on:
- Enlarging the Corpus: Expanding the sound dataset with multilingual data to improve generalizability.
- Adaptive Prompts: Developing adaptive prompts that personalize both acoustic and linguistic priors for individual users.
- User Study: Conducting a within-subjects user study to evaluate CAFA against traditional audiologist-user conversations, measuring task completion time, cognitive load, usability, and self-reported listening benefit.
Summary updated