A comparison study on patient-psychologist voice diarization

Rachid Riad, Hadrien Titeux, Laurie Lemoine, Justine Montillot, Agnes Sliwinski, Jennifer Hamet Bagnou, Xuan-Nga Cao, Anne-Catherine Bachoud-Lévi, Emmanuel DupouxSLPAT @ ACL 2022 · DOI

Introduction

In clinical settings, speech patterns—such as pauses, turn-taking behavior, and speech rate—serve as vital "digital biomarkers" for neurodegenerative conditions like Huntington’s Disease (HD). However, manual annotation of long clinical interviews is labor-intensive and requires specialized expertise. This study investigates automated "who spoke when" (diarization) techniques, specifically comparing general speaker identification against role-specific recognition in a psychologist-patient context.

Methods

The researchers evaluated two primary architectural approaches to automate diarization in clinical interviews:

  1. Speaker Role Recognition (SRR): An end-to-end neural network that processes raw audio via SincNet filters and Bi-LSTM layers. It is trained to directly classify segments as "Psychologist" or "Patient," capturing role-specific prosodic and linguistic traits rather than individual voice identities.
  2. Speaker Enrollment Protocol (SEP): A traditional multi-stage pipeline involving Voice Activity Detection (VAD), Speaker Change Detection (SCD), and speaker matching using X-vector embeddings against a pre-enrolled voice template of the speaker.
Comparison of the Speaker Enrollment Protocol (SEP) pipeline and the end-to-end Speaker Role Recognition (SRR) model.

To ensure generalizability, a "meta-split" strategy was used, testing models on entirely unseen speakers to prevent the networks from simply memorizing individual voices from the training set.

Datasets

The study utilized a specialized dataset of 94 clinical interviews from Henri-Mondor Hospital, France. The participants were categorized into three groups: Healthy Controls (22), Pre-manifest gene carriers (11), and Manifest HD patients (33). Models were pre-trained on large-scale public datasets like DIHARD and VoxCeleb2 before being fine-tuned on this specialized clinical data to account for hospital acoustics and dysarthric speech patterns.

Key Results

The experimental results highlighted the superiority of end-to-end modeling over traditional pipelines in clinical environments.

  • Performance Lead: The end-to-end SRR model achieved the lowest Identification Error Rate (IER) of 19.5%. In contrast, the best-performing SEP pipeline (utilizing fine-tuned embeddings and retrained VAD/SCD) reached an IER of 23.6%.
  • Error Propagation: Multi-stage pipelines suffered from "error accumulation," where inaccuracies in early stages (like VAD) negatively impacted subsequent speaker identification. The end-to-end approach avoids this by optimizing for the final task directly.
  • Fine-tuning Necessity: Models trained only on general datasets (e.g., YouTube or news) performed poorly. Domain-specific fine-tuning was essential to handle the unique acoustic environment of a hospital and the specific speech patterns of patients with neurodegenerative disorders.
Identification Error Rate (IER) comparison across different enrollment methods and the SRR model.

Clinical Implications: The "Silence Ratio"

Beyond technical accuracy, the study validated the clinical utility of automated diarization. The system successfully extracted the Ratio of Silence, a metric that typically increases as Huntington’s Disease progresses. The automatically derived silence ratio showed a significant correlation with ground truth annotations (r = 0.579, p = 0.009), suggesting it can serve as a reliable automated biomarker for monitoring disease state and progression.

Conclusion

While the models excel at providing global interview summaries, precision in fine-grained turn-taking remains a challenge, often due to background noise in hospital environments. Future work aims to refine these temporal detections and explore direct prediction of clinical scores, such as the Total Motor Score, to enable non-invasive, home-based patient monitoring.

Summary updated

BibTeX
@inproceedings{riad2022comparison,
  author = {Rachid Riad and Hadrien Titeux and Laurie Lemoine and Justine Montillot and Agnes Sliwinski and Jennifer Hamet Bagnou and Xuan-Nga Cao and Anne-Catherine Bachoud-Lévi and Emmanuel Dupoux},
  title = {{A comparison study on patient-psychologist voice diarization}},
  booktitle = {SLPAT @ ACL},
  year = {2022},
  doi = {10.18653/v1/2022.slpat-1.4},
}
Figure