Looking for relevant features for speaker role recognition

Benjamin Bigot, Julien Pinquier, Isabelle Ferrané, Régine André-ObrechtInterspeech 2010

TL;DR

This paper tackles the challenge of automatically identifying speaker roles (like anchor, journalist, or guest) in audio recordings, moving beyond traditional broadcast news to more heterogeneous content. They introduce a novel, language-independent method that leverages low-level acoustic, temporal, and prosodic features, achieving an impressive 85% accuracy by employing a hierarchical classification strategy and a robust feature selection technique.

What Problem Does This Paper Address?

Identifying who is speaking and what role they play in audio and video content is crucial for tasks like information retrieval, summarization, and understanding social dynamics within conversations. While previous work has focused on well-structured broadcast news with clear speaker roles, this paper aims to develop a more generalizable method. The challenge lies in creating a system that can accurately assign roles across diverse audio content (like radio shows with varying structures and durations) without relying on transcriptions or specific language models, and to do so in a language-independent manner. They also aim to refine existing role categories to better capture the nuances of speaker interactions.

Methods

The core of their approach is a hierarchical classification system built upon automatic speaker segmentation (diarization). Instead of directly classifying into 5 roles, they break it down into a series of binary classifications.

  1. Speaker Diarization: They start with an existing speaker diarization tool that segments the audio into turns for each speaker.
  2. Punctual vs. Non-Punctual Separation: A key innovation is distinguishing "punctual" speakers (appearing in only one segment) from "non-punctual" ones. This is a crucial step for handling the diversity of roles.
  3. Hierarchical Classification:
    • Anchor vs. Non-Anchor: The first binary classification distinguishes the dominant anchor from all other speakers.
  4. Journalist vs. Other: For non-anchors, a second classification separates professional journalists from other participants.
  5. Punctual Journalist vs. Punctual Other: Finally, for punctual speakers, a distinction is made between punctual journalists and punctual others.
  6. Feature Extraction: They extract a rich set of 36 low-level features, including:
  7. Temporal and Acoustic Features: These capture segment lengths, inter-segment durations, speech activity, and signal power characteristics.
  8. Prosodic Features: These include pitch (fundamental frequency) variations and characteristics of vocalic segments and silences.
  9. Feature Selection: Crucially, before each classification step, they employ a Sequential Backward Feature Selection (SBFS) method. This technique iteratively removes features that contribute least to accuracy, identifying the most discriminative features for each specific classification task. This is a significant departure from previous work that relied on dimensionality reduction techniques like PCA.
  10. Classification: They experiment with Gaussian Mixture Models (GMM), K-Nearest Neighbors (K-nn), and Support Vector Machines (SVM) with linear and RBF kernels.

Datasets

  • DEV-ref: A development corpus of 435 minutes of audio from 20 radio documents (news programs and talk-shows) recorded in June-July 2007. This was used for training the models and estimating feature characteristics.
  • TEST-auto: A test corpus of 440 minutes of audio from 26 radio documents recorded in January-February 2008. This corpus was used for evaluation. The ground truth for speaker diarization was provided by the ESTER2 campaign organizers.

The paper defines 5 speaker roles: Anchor (A), Punctual Journalist (PJ), Journalist (J), Punctual Other (PO), and Other (O). The dataset statistics for DEV-ref and TEST-auto are provided for these roles.

Key Insights and Ideas

  • Low-level features are powerful: The paper strongly suggests that acoustic, temporal, and prosodic features, even without relying on transcriptions or semantic content, are sufficient for speaker role recognition, especially when combined with effective feature selection.
  • Hierarchical classification is effective: Breaking down the complex 5-role problem into a series of binary classifications significantly improves performance and manageability.
  • Punctual/Non-Punctual distinction is key: Differentiating between speakers who appear briefly and those who are more consistently present is a critical step for handling diverse audio content and accurately assigning roles.
  • Feature selection is superior to dimensionality reduction: The SBFS method, by identifying the most relevant features for each specific task, leads to better accuracy (a 13% gain compared to their previous PCA-based approach) and potentially more interpretable models.
  • Anchor role is relatively easy: The Anchor role can be recognized with high accuracy using a small subset of temporal and prosodic features.
  • Journalist/Other is challenging: Distinguishing between non-punctual Journalists and Others is more difficult and requires a larger set of features, with errors often stemming from speaker diarization inaccuracies.
  • Punctual roles benefit from prosody: The distinction between Punctual Journalists and Punctual Others seems to be well captured by prosodic features, possibly reflecting differences in spontaneous vs. prepared speech.

Key Results

  • Overall Accuracy: The proposed system achieves an overall accuracy of 85.6% ± 8% for correctly labeling speaker roles on the TEST-auto corpus. This is a significant improvement over their previous work (72%).
  • Anchor/Non-Anchor: Achieved up to 96% accuracy using a single Gaussian Model with all 36 features, and 92% with a linear SVM using only 14 features.
  • Journalist/Other: The best accuracy reached was 74% with a linear SVM, using 26 features.
  • Punctual Journalist/Punctual Other: The linear SVM classifier achieved 88.9% accuracy with 10 selected features.

Limitations and Future Work

  • Speaker Diarization Errors: The paper acknowledges that errors in the initial speaker diarization system can lead to misclassifications, particularly in distinguishing punctual speakers.
  • Complexity of Journalist/Other: Classifying non-punctual Journalists and Others remains a more difficult task, requiring more features and still susceptible to diarization errors.
  • Future Work: The authors plan to leverage these speaker role recognition results as prior knowledge for automatic speech recognition systems to select more appropriate language models, aiming to improve the overall speech recognition performance. They also intend to further explore these results for analyzing speaker interactions and content structuring.

Summary updated

BibTeX
@inproceedings{bigot2010looking,
  author = {Benjamin Bigot and Julien Pinquier and Isabelle Ferrané and Régine André-Obrecht},
  title = {{Looking for relevant features for speaker role recognition}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure