Abstract
Extracting patient medical conditions from code-switched clinical spoken dialogues is challenging due to rapid turn-taking and highly overlapped speech. We present a robust system evaluated on the DISPLACE-M dataset of real-world Hinglish medical conversations. We propose an End-to-End Neural Diarization with Vector Clustering approach (EEND-VC) to accurately resolve dense and speaker overlaps in Doctor-Patient Conversations (DoPaCo). For transcription, we adapt a Qwen3 ASR model via domain-specific fine-tuning, Devanagari script normalization, and dialogue-level LLM error correction, achieving an 18.59% tcpWER. We benchmark open and proprietary LLMs on medical condition extraction, comparing our text-based cascade system against a multimodal End-to-End (E2E) audio framework. While proprietary E2E models set the performance ceiling, our open cascaded architecture is highly competitive, as it achieved first place out of 25 participants in the DISPLACE-M challenge.
Introduction
In the chaotic environment of rural Indian health clinics, conversations between community health workers (ASHAs) and patients are rarely tidy. These interactions are characterized by "Hinglish" code-switching (fluidly flipping between Hindi and English), rapid turn-taking with frequent overlapped speech, and significant acoustic noise from far-field recordings.
Standard Automatic Speech Recognition (ASR) systems often fail in these settings because they assume single-speaker clarity or struggle with the linguistic fluidity of code-switched speech. To extract medical conditions accurately, a system must first resolve who is speaking (Diarization) and then accurately transcribe their specific contributions (Speaker-Attributed ASR).
System Architecture
The researchers developed a modular "cascade" pipeline consisting of three main stages designed to handle the complexities of clinical dialogues.
- Diarization (EEND-VC): The system utilizes End-to-End Neural Diarization with Vector Clustering. Unlike older hybrid systems that assume sequential speech, this model natively handles simultaneous speech. The team used
w2v-bert2.0as a backbone, finding it significantly more robust for Hindi than standard English-centric models. - Speaker-Attributed ASR (SA-ASR): They adapted the Qwen3-ASR-1.7B model, conditioning it to only transcribe segments identified by the diarization module as "active" speech from the doctor or patient. This effectively filters out environmental noise and irrelevant background chatter.
- Medical Condition Extraction: Once the text is generated, it passes through a Large Language Model (LLM) for error correction and final extraction. A key technical contribution here is Devanagari script normalization, which ensures the ASR scoring isn't unfairly penalized for inconsistent Unicode formatting of Hindi words.
Datasets and Training
The system was evaluated on the DISPLACE-M dataset, containing ~35 hours of real-world primary healthcare interactions in rural India. To ensure the diarization model could generalize across acoustic environments, it was pre-trained on a massive mix of datasets (VoxConverse, AMI, AliMeeting, etc.) totaling over 900 hours. For ASR adaptation, the team aggregated ~1,800 hours of Hindi speech to fine-tune the model for the specific medical domain and regional dialects.
Key Results
The system achieved 1st place out of 25 participants in the DISPLACE-M challenge. A major driver of this success was the incremental improvement in ASR performance through domain-specific tuning and error correction.
As shown in the results, the final system achieved an 18.59% tcpWER (temporal-constrained phoneme Word Error Rate), a substantial improvement over the 26.78% baseline. In the medical extraction task, their setup proved that open-weight models can effectively handle complex clinical extraction if the upstream transcription is high-quality.
Insights and Discussion
Several critical findings emerged from the development process:
- Diarization is the Bottleneck: Upgrading ASR alone didn't help if diarization was poor. High-quality diarization combined with baseline ASR actually performed better than the reverse.
- Domain Specificity Trumps Model Size: For ASR, fine-tuning on domain-specific medical and Hindi data provided a much larger performance boost than simply using a larger general-purpose model.
- The "Lost-in-the-Middle" Effect: When using LLMs for error correction, providing the full dialogue often degraded performance. Success was found by extracting "contrastive snippets"—short windows of text where the ASR likely made a mistake—and asking the LLM to fix those specifically.
- E2E vs. Cascade: While proprietary end-to-end models (like Gemini 3 Pro processing raw audio) currently set the performance ceiling, a well-tuned open-source cascade is highly competitive and offers superior transparency and modularity.
Conclusion
This research demonstrates that synergizing robust diarization with domain-adapted ASR can overcome the challenges of extracting information from noisy, code-switched medical dialogues. While the cascaded approach is susceptible to error propagation, its modularity allows for targeted improvements at each stage. Future work will focus on tighter multimodal integration to preserve acoustic cues like tone or hesitation that are currently lost during text conversion.
Summary updated