Doctor or Patient? Synergizing Diarization and ASR for Code-Switched Hinglish Medical Conditions Extraction

Séverin Baroudi, Yanis Labrak, Shashi Kumar, Joonas Kalda, Sergio Burdisso, Paweł Cyrta, Juan Ignacio Alvarez-Trejos, Petr Motlíček, Hervé Bredin, Ricard MarxerarXiv 2026 · DOI · arXiv

Abstract

Extracting patient medical conditions from code-switched clinical spoken dialogues is challenging due to rapid turn-taking and highly overlapped speech. We present a robust system evaluated on the DISPLACE-M dataset of real-world Hinglish medical conversations. We propose an End-to-End Neural Diarization with Vector Clustering approach (EEND-VC) to accurately resolve dense and speaker overlaps in Doctor-Patient Conversations (DoPaCo). For transcription, we adapt a Qwen3 ASR model via domain-specific fine-tuning, Devanagari script normalization, and dialogue-level LLM error correction, achieving an 18.59% tcpWER. We benchmark open and proprietary LLMs on medical condition extraction, comparing our text-based cascade system against a multimodal End-to-End (E2E) audio framework. While proprietary E2E models set the performance ceiling, our open cascaded architecture is highly competitive, as it achieved first place out of 25 participants in the DISPLACE-M challenge.

Introduction

In the chaotic environment of rural Indian health clinics, conversations between community health workers (ASHAs) and patients are rarely tidy. These interactions are characterized by "Hinglish" code-switching (fluidly flipping between Hindi and English), rapid turn-taking with frequent overlapped speech, and significant acoustic noise from far-field recordings.

Standard Automatic Speech Recognition (ASR) systems often fail in these settings because they assume single-speaker clarity or struggle with the linguistic fluidity of code-switched speech. To extract medical conditions accurately, a system must first resolve who is speaking (Diarization) and then accurately transcribe their specific contributions (Speaker-Attributed ASR).

System Architecture

The researchers developed a modular "cascade" pipeline consisting of three main stages designed to handle the complexities of clinical dialogues.

The proposed cascaded pipeline for medical condition extraction, featuring EEND-VC diarization, speaker-attributed ASR, and LLM-based extraction.
  • Diarization (EEND-VC): The system utilizes End-to-End Neural Diarization with Vector Clustering. Unlike older hybrid systems that assume sequential speech, this model natively handles simultaneous speech. The team used w2v-bert2.0 as a backbone, finding it significantly more robust for Hindi than standard English-centric models.
  • Speaker-Attributed ASR (SA-ASR): They adapted the Qwen3-ASR-1.7B model, conditioning it to only transcribe segments identified by the diarization module as "active" speech from the doctor or patient. This effectively filters out environmental noise and irrelevant background chatter.
  • Medical Condition Extraction: Once the text is generated, it passes through a Large Language Model (LLM) for error correction and final extraction. A key technical contribution here is Devanagari script normalization, which ensures the ASR scoring isn't unfairly penalized for inconsistent Unicode formatting of Hindi words.

Datasets and Training

The system was evaluated on the DISPLACE-M dataset, containing ~35 hours of real-world primary healthcare interactions in rural India. To ensure the diarization model could generalize across acoustic environments, it was pre-trained on a massive mix of datasets (VoxConverse, AMI, AliMeeting, etc.) totaling over 900 hours. For ASR adaptation, the team aggregated ~1,800 hours of Hindi speech to fine-tune the model for the specific medical domain and regional dialects.

Key Results

The system achieved 1st place out of 25 participants in the DISPLACE-M challenge. A major driver of this success was the incremental improvement in ASR performance through domain-specific tuning and error correction.

ASR performance (tcpWER) across different system configurations, showing the impact of fine-tuning and error correction.

As shown in the results, the final system achieved an 18.59% tcpWER (temporal-constrained phoneme Word Error Rate), a substantial improvement over the 26.78% baseline. In the medical extraction task, their setup proved that open-weight models can effectively handle complex clinical extraction if the upstream transcription is high-quality.

Insights and Discussion

Several critical findings emerged from the development process:

  • Diarization is the Bottleneck: Upgrading ASR alone didn't help if diarization was poor. High-quality diarization combined with baseline ASR actually performed better than the reverse.
  • Domain Specificity Trumps Model Size: For ASR, fine-tuning on domain-specific medical and Hindi data provided a much larger performance boost than simply using a larger general-purpose model.
  • The "Lost-in-the-Middle" Effect: When using LLMs for error correction, providing the full dialogue often degraded performance. Success was found by extracting "contrastive snippets"—short windows of text where the ASR likely made a mistake—and asking the LLM to fix those specifically.
  • E2E vs. Cascade: While proprietary end-to-end models (like Gemini 3 Pro processing raw audio) currently set the performance ceiling, a well-tuned open-source cascade is highly competitive and offers superior transparency and modularity.

Conclusion

This research demonstrates that synergizing robust diarization with domain-adapted ASR can overcome the challenges of extracting information from noisy, code-switched medical dialogues. While the cascaded approach is susceptible to error propagation, its modularity allows for targeted improvements at each stage. Future work will focus on tighter multimodal integration to preserve acoustic cues like tone or hesitation that are currently lost during text conversion.

Summary updated

Abstract

Extracting patient medical conditions from code-switched clinical spoken dialogues is challenging due to rapid turn-taking and highly overlapped speech. We present a robust system evaluated on the DISPLACE-M dataset of real-world Hinglish medical conversations. We propose an End-to-End Neural Diarization with Vector Clustering approach (EEND-VC) to accurately resolve dense and speaker overlaps in Doctor-Patient Conversations (DoPaCo). For transcription, we adapt a Qwen3 ASR model via domain-specific fine-tuning, Devanagari script normalization, and dialogue-level LLM error correction, achieving an 18.59% tcpWER. We benchmark open and proprietary LLMs on medical condition extraction, comparing our text-based cascade system against a multimodal End-to-End (E2E) audio framework. While proprietary E2E models set the performance ceiling, our open cascaded architecture is highly competitive, as it achieved first place out of 25 participants in the DISPLACE-M challenge. All implementations are publicly released.
BibTeX
@misc{baroudi2026doctor,
  author = {Séverin Baroudi and Yanis Labrak and Shashi Kumar and Joonas Kalda and Sergio Burdisso and Paweł Cyrta and Juan Ignacio Alvarez-Trejos and Petr Motlíček and Hervé Bredin and Ricard Marxer},
  title = {{Doctor or Patient? Synergizing Diarization and ASR for Code-Switched Hinglish Medical Conditions Extraction}},
  year = {2026},
  eprint = {2603.06373},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.06373},
}
Figure