A comparison of neural network feature transforms for speaker diarization

Sree Harsha Yella, Andreas StolckeInterspeech 2015

TL;DR

This paper explores using artificial neural networks (ANNs) to extract better speech features for speaker diarization, outperforming traditional methods. By training ANNs to compare or classify speakers, they achieve significant reductions in diarization error, especially when combining these new features with existing ones.

What Problem Does This Paper Address?

Speaker diarization is the task of identifying "who spoke when" in an audio recording, without prior knowledge of the speakers. This is crucial for tasks like transcribing multi-party conversations, analyzing meetings, or organizing call center recordings. Traditional diarization systems rely on short-term spectral features like MFCCs, which capture general speech characteristics but aren't optimized for distinguishing between speakers. Factors like background noise, room acoustics, and conversational speaking styles make this task particularly challenging. While techniques like i-vectors and LDA have shown promise, they haven't fully solved the problem, especially for complex scenarios like meeting room recordings. This paper aims to improve speaker diarization by leveraging the discriminative power of neural networks to learn more effective speech features.

Methods

The authors investigate three main types of ANN architectures for feature extraction:

  • Shallow Speaker Comparison ANN (spkr-com): This network is trained to determine if two speech segments belong to the same speaker or different speakers. It uses a bottleneck layer to learn a common transform for speech features.
  • Deep Speaker Comparison ANN (Dspkr-com): An extension of the shallow network, this architecture adds an extra hidden layer before the bottleneck. This allows for a nonlinear feature transform, potentially capturing more complex speaker characteristics.
  • Speaker Classification ANN (spkr-class): This network is trained to classify speech segments into specific speaker identities. This approach, previously used for speaker recognition, is adapted here for diarization.
  • Autoencoder ANN (autoen): As a baseline, an autoencoder is used to learn a compressed representation of speech features through reconstruction.

For all ANN-based methods, the features are extracted from the activations of a hidden layer (the "bottleneck" layer) before the final output layer. These extracted features are then used in a standard HMM/GMM speaker diarization system. The paper also explores combining these ANN-derived features with traditional MFCC features by weighting their respective likelihoods in the diarization model.

Datasets

The experiments utilize data from three well-established corpora:

  • AMI: Used for both training and testing, with disjoint speakers between the sets.
  • ICSI: A portion is used for development (tuning parameters), and the rest for testing.
  • NIST-RT (2006, 2007, 2009): Combined evaluation sets used for testing.

The paper notes that the AMI corpus was used for ANN training, and the test sets were chosen to represent varying degrees of match to this training condition.

Key Insights and Ideas

  • Discriminative Training is Key: ANNs trained specifically for speaker-related tasks (comparison or classification) yield better diarization features than those trained for general reconstruction (autoencoders). This highlights the importance of directly optimizing for speaker discrimination.
  • Going Deeper Helps: Adding an extra hidden layer to the speaker comparison network (Dspkr-com) significantly improves performance, suggesting that deeper architectures can learn more nuanced and discriminative features.
  • Complementary Information: ANN-derived features, even if not outperforming MFCCs on their own, can significantly improve diarization when combined with MFCCs. This suggests that ANNs capture information that is complementary to traditional spectral features.
  • Training Data Matters: While testing with heterogeneous speech (non-speech, other speakers) is realistic, training ANNs on pure, speaker-homogeneous speech segments appears to yield better results, even if it creates a mismatch with test conditions.

Key Results

  • The deep speaker comparison network (Dspkr-com) combined with MFCCs (M+Dspkr-com) consistently reduced speaker error across all test sets compared to using MFCCs alone.
  • The speaker classification ANN (spkr-class) combined with MFCCs (M+spkr-class) achieved the largest relative reduction in speaker error, a 34% decrease on the ICSI test set.
  • On the NIST-RT data processed with automatic speech activity detection (SAD), the M+Dspkr-com combination reduced speaker error by 20% relative to the MFCC baseline.
  • Autoencoder features did not show significant gains when combined with MFCCs, indicating their limitations in capturing speaker-discriminative information.

Limitations and Future Work

  • Limitations: The paper notes that none of the ANN features, when used alone, consistently outperformed MFCCs. The autoencoder features, in particular, did not show complementary benefits when combined with MFCCs. The training of ANNs on speaker-homogeneous data, while effective, creates a mismatch with real-world test conditions.
  • Future Work: The authors suggest that further exploration of ANN architectures and training criteria could lead to even greater improvements. The success of combining ANN features with MFCCs suggests that multi-stream approaches are a promising direction. Investigating how to better handle heterogeneous speech during training could also be a valuable area for future research.

Summary updated

BibTeX
@inproceedings{yella2015comparison,
  author = {Sree Harsha Yella and Andreas Stolcke},
  title = {{A comparison of neural network feature transforms for speaker diarization}},
  booktitle = {Interspeech},
  year = {2015},
}
Figure