Supervised Speaker Diarization Using Random Forests: A Tool for Psychotherapy Process Research

Lukas Fürer, Nathalie Schenk, Volker Röth, Martin Steppan, Klaus Schmeck, Ronan ZimmermannFrontiers in Psychology 2020 · DOI

Abstract

Speaker diarization is the practice of determining who speaks when in audio recordings. Psychotherapy research often relies on labor intensive manual diarization. Unsupervised methods are available but yield higher error rates. We present a method for supervised speaker diarization based on random forests. It can be considered a compromise between commonly used labor-intensive manual coding and fully automated procedures. The method is validated using the EMRAI synthetic speech corpus and is made publicly available. It yields low diarization error rates (M: 5.61%, STD: 2.19). Supervised speaker diarization is a promising method for psychotherapy research and similar fields.

Introduction

In psychotherapy research, the study of interpersonal synchrony—the way patients and therapists mirror each other's verbal and non-verbal behaviors—is essential for predicting therapeutic outcomes. A prerequisite for this analysis is accurate speaker diarization: the process of identifying exactly who is speaking at any given moment.

Currently, researchers face a difficult choice. Manual diarization is highly accurate but prohibitively labor-intensive, often requiring many times the duration of the actual session to complete. Conversely, unsupervised automated methods are fast but prone to high error rates (frequently exceeding 10–20% in clinical settings), which can introduce significant noise into synchrony data. This paper proposes a supervised approach using Random Forest classifiers to bridge this gap, offering near-manual accuracy with a significantly reduced human-in-the-loop investment.

Methodology

The proposed pipeline treats diarization as a dyad-specific classification problem. Rather than a universal model, a custom classifier is trained for each unique patient-therapist pair.

Workflow and Feature Extraction

The process begins with a "manual seed," where a human labels the first 10 minutes of a session. After automatic Voice Activity Detection (VAD) filters out non-speech segments, the audio is divided into 0.1-second windows. For each window, 35 standard audio features are extracted, including Mel-frequency cepstral coefficients (MFCCs), spectral flux, and entropy. Crucially, the authors include the HF500 ratio (high-to-low spectral energy), which provides a stable "fingerprint" of voice quality.

A Random Forest ensemble of 500 decision trees is then trained on this 10-minute sample. This model is used to diarize the remainder of the session and subsequent sessions involving the same dyad. Random Forests were selected for their robustness against multicollinearity and their ability to perform well without the massive datasets required by deep learning architectures.

Validation Dataset

The method was validated using the EMRAI Synthetic Diarization Corpus, consisting of 107 synthetic dialogues derived from the LibriSpeech dataset. Synthetic data provides a "perfect" ground truth, allowing for precise measurement of speaker confusion without the interference of real-world overlapping speech or environmental noise.

Results

The supervised Random Forest approach demonstrated high reliability and accuracy across the test corpus. The system achieved a mean Diarization Error Rate (DER) of 5.61%. Analysis of the error components revealed that the majority of the error was attributable to silence detection (missed speech or false alarms), while the actual Speaker Error (SpE) was remarkably low at 2.50%.

Table 1: Performance metrics across all dyads, same-sex dyads, and different-sex dyads.

As shown in Table 1, the system's performance varied by dyad composition. Different-sex dyads achieved a lower DER (5.01%) compared to same-sex dyads (6.48%), suggesting that spectral similarities between speakers of the same sex remain a challenge for the classifier.

Discussion: The Out-of-Bag Proxy

A significant practical advantage of the Random Forest approach is the utility of the "out-of-bag" (OOB) error rate. The researchers found a strong correlation ($r = 0.85$) between the model's internal OOB error and the actual speaker error on the test set.

Figure 1: Correlation between the internal Out-of-Bag (OOB) error and the actual Dyadic Speaker Error.

This correlation, illustrated in Figure 1, allows researchers to estimate the reliability of a specific diarization without needing a separate manually-labeled test set. If the OOB error is high, the researcher is alerted that the diarization for that specific dyad may require further manual review.

Conclusion

Supervised speaker diarization via Random Forests offers a viable "middle ground" for clinical research. While it requires an initial manual investment and currently struggles with overlapping speech, its accuracy and the reliability of the OOB error proxy make it a powerful tool for psychotherapy process research. Future work should focus on transitioning the tool from MATLAB to more accessible platforms and improving the handling of "overtalk" and real-world acoustic environments.

Summary updated

Abstract

Speaker diarization is the practice of determining who speaks when in audio recordings. Psychotherapy research often relies on labor intensive manual diarization. Unsupervised methods are available but yield higher error rates. We present a method for supervised speaker diarization based on random forests. It can be considered a compromise between commonly used labor-intensive manual coding and fully automated procedures. The method is validated using the EMRAI synthetic speech corpus and is made publicly available. It yields low diarization error rates (M: 5.61%, STD: 2.19). Supervised speaker diarization is a promising method for psychotherapy research and similar fields.
BibTeX
@inproceedings{furer2020supervised,
  author = {Lukas Fürer and Nathalie Schenk and Volker Röth and Martin Steppan and Klaus Schmeck and Ronan Zimmermann},
  title = {{Supervised Speaker Diarization Using Random Forests: A Tool for Psychotherapy Process Research}},
  booktitle = {Frontiers in Psychology},
  year = {2020},
  doi = {10.3389/fpsyg.2020.01726},
}
Figure