Speaker change detection and tracking in real-time news broadcasting analysis

Lie Lu, HongJiang ZhangACM Multimedia 2002 · DOI

TL;DR

This paper introduces a novel, real-time, unsupervised system for detecting speaker changes and tracking speakers in news broadcasts. It tackles the challenge of unknown speaker identities and numbers by employing a two-step detection process and a Bayesian fusion of audio features for robust performance.

What Problem Does This Paper Address?

Indexing and analyzing broadcast news content often requires understanding who is speaking and when they change. Traditional speaker recognition systems typically assume prior knowledge of speaker identities. However, in real-time scenarios like live news, this information is unavailable. The challenge is to segment continuous speech streams into segments belonging to different speakers (speaker segmentation/change detection) and then identify those speakers (speaker tracking) without any pre-existing knowledge of who the speakers are or how many there will be. This is crucial for applications like content-based retrieval, meeting indexing, and speaker adaptation.

Methods

The proposed system uses a two-step approach for speaker change detection:

  • Potential Change Detection: This step involves segmenting the audio into short, overlapping sub-segments (3 seconds with 2.5-second overlap). For each sub-segment, features like Linear Spectral Pairs (LSP), Mel-Frequency Cepstrum Coefficients (MFCC), and pitch are extracted. A "divergence shape distance" is calculated between consecutive sub-segments. If this distance exceeds an automatically adjusted threshold, a potential speaker change is flagged.
  • Refinement: To reduce false positives from the initial detection, a Bayesian decision engine is employed. This engine fuses the distances derived from LSP, MFCC, and pitch using Bayesian inference to make a more reliable decision about whether a true speaker change has occurred.

Speaker tracking is then performed based on these detected change points. The system uses an incremental approach to build speaker models. When a potential speaker change is detected, the system attempts to identify the new speaker by comparing the current segment's features against a database of existing speaker models. If a match is found, the model is updated; otherwise, a new speaker is registered.

A key aspect of their method is the Bayesian Fusion of Multiple Audio Features. Instead of relying on a single feature, they combine LSP, MFCC, and pitch to achieve more robust results, acknowledging that different features perform better in different contexts. They also use Cepstral Mean Subtraction (CMS) to mitigate channel and environment variations.

For speaker model updating, they use a quasi-Gaussian Mixture Model (GMM) approach. This method is designed to be computationally efficient for real-time processing, unlike traditional EM algorithms for GMMs.

Datasets

The evaluations were performed on the Hub-4 1997 English Broadcast News Speech Database. Specifically, they used news broadcasts from ABC World News Tonight and CNN Headline News, totaling about 10 hours of audio. Half of this data was used for training and the other half for testing. Each test file was approximately 30 minutes long, containing around 30 speakers and 60-80 speaker changes. The ground truth was obtained from accompanying transcripts.

Key Insights and Ideas

  • Unsupervised Real-Time is Key: The paper emphasizes the necessity of unsupervised methods for real-time applications where prior speaker knowledge is absent.
  • Feature Fusion for Robustness: Combining LSP, MFCC, and pitch via Bayesian fusion significantly improves performance compared to using individual features, especially in challenging broadcast environments.
  • Incremental Model Building: The quasi-GMM approach allows for gradual model refinement as more data becomes available, balancing accuracy with real-time constraints.
  • Automatic Thresholding: Dynamically adjusting the speaker change detection threshold based on recent history helps adapt to varying audio conditions.
  • Trade-off Between Accuracy and Delay: The paper highlights that more accurate speaker identification (using more data) comes at the cost of increased decision delay, a critical factor in real-time systems.

Key Results

  • Speaker Segmentation: Achieved a recall of over 85% with a false alarm rate of 15% for long speaker segments (>3s). For short segments (<3s), recall dropped to 37.6%.
  • Boundary Accuracy: Approximately 70% of detected speaker change boundaries were within 1 second of the true boundary, and 86.6% were within 2 seconds.
  • Speaker Tracking: The "Bound" method (identifying speakers after the next real boundary is confirmed) yielded the best performance, showing a significant recall advantage (up to 20% higher at a 0.1 false alarm rate) compared to "Instant" or "Change" methods.
  • Computational Complexity: The algorithm can complete segmentation and tracking in about 15% of the audio clip's duration on a Pentium III processor, meeting real-time requirements.

Limitations and Future Work

  • Short Speaker Segments: The algorithm's performance is significantly degraded for speaker segments shorter than 3 seconds, which is the minimum unit for model estimation.
  • False Alarms from Non-Speech: Misclassification of non-speech segments (like laughs or wind bursts) as speech leads to a substantial number of false alarms in speaker change detection.
  • Environment/Channel Mismatch: While CMS is used, the paper acknowledges that compensation for environmental and channel variations is still insufficient, leading to potential misidentification of the same speaker in different conditions.
  • Future Work: The authors suggest focusing on improving model estimation from limited training data, addressing false alarms caused by non-speech segments, and enhancing compensation for environmental and channel variations. They also note that the system's performance can be improved by reducing the number of speakers in a given segment.

Summary updated

BibTeX
@inproceedings{lu2002speaker,
  author = {Lie Lu and HongJiang Zhang},
  title = {{Speaker change detection and tracking in real-time news broadcasting analysis}},
  booktitle = {ACM Multimedia},
  year = {2002},
  doi = {10.1145/641007.641127},
}
Figure