Speaker Augmentation and Bandwidth Extension for Deep Speaker Embedding

Hitoshi Yamamoto, Kong Aik Lee, Koji Okabe, Takafumi KoshinakaInterspeech 2019 · OpenReview

Abstract

This paper investigates a novel data augmentation approach to train deep neural networks (DNNs) used for speaker embedding. By generating new speakers via voice conversion (speed perturbation), we expand the coverage of speakers in the embedding space, improving the model's discriminative power. We also utilize bandwidth extension (BWE) to bridge the gap between narrowband and wideband audio. Experiments on the Speakers in the Wild (SITW) and NIST SRE datasets demonstrate significant reductions in detection costs, with speaker augmentation alone providing a 17.9% improvement.

Introduction

Deep learning models for speaker recognition, such as the x-vector architecture, rely on learning highly discriminative features to separate unique voices. To achieve high performance, these models require training on a massive variety of speakers. However, collecting and labeling data from tens of thousands of unique individuals is expensive and difficult. While standard "audio augmentation" (adding noise or reverb) helps models handle environmental variability, it does not increase the number of speaker identities the model learns to distinguish. This paper addresses this "identity bottleneck" by introducing speaker augmentation and bandwidth extension to artificially expand the training set.

Methods

The authors propose two primary strategies to enhance speaker embedding training:

  1. Speaker Augmentation via Voice Conversion: Unlike traditional augmentation that keeps the speaker label the same, this method uses speed perturbation (changing pitch and tempo via resampling) to create a "new" voice. These perturbed versions are assigned unique speaker IDs. For example, by creating two copies (at 0.9x and 1.1x speed) of $N$ original speakers, the model is trained to classify $3N$ distinct speaker identities.
  2. DNN-based Bandwidth Extension (BWE): To utilize both narrowband (8kHz) and wideband (16kHz) data, a pre-processing DNN takes 23 log mel-filterbank coefficients (MFBs) from narrowband speech and estimates the missing 9 high-frequency MFBs. This produces a full 32-MFB wideband-style feature, allowing a single wideband x-vector extractor to be trained on a much larger, mixed-bandwidth pool of speakers.
Overview of the speaker embedding system incorporating speaker augmentation and mixed-bandwidth training.

Key Insights and Ideas

  • Identity vs. Variability: Traditional augmentation targets within-class variability (robustness to noise). Speaker augmentation targets between-class discriminability (telling similar people apart).
  • Virtual Speakers are Effective: A surprising finding is that "virtual" speakers created by simple signal processing are as effective for training as adding real speakers from a different corpus.
  • The Power of Scale: This approach brings the "massive identity" philosophy common in face recognition to the audio domain without the prohibitive cost of data collection.
  • Complementary Techniques: Bandwidth extension allows the model to access more data sources, while speaker augmentation multiplies the identities within those sources.

Key Results

The effectiveness of these methods was validated on the Speakers in the Wild (SITW) and NIST SRE 2018 benchmarks:

  • SITW Performance: Speaker augmentation reduced the minimum Detection Cost Function (mDCF) from 0.330 to 0.271—a 17.9% improvement. The Equal Error Rate (EER) also dropped from 3.28% to 2.73%.
  • NIST SRE 2018: The combination of mixed-bandwidth training, BWE, and speaker augmentation yielded the best results, achieving an mDCF of 0.151 and an EER of 2.18%, significantly outperforming the wideband baseline.
Performance metrics on NIST SRE 2018 for various system configurations.

Limitations and Future Work

While effective, the authors used basic speed perturbation for speaker augmentation. More sophisticated voice conversion or synthesis techniques could potentially create even more diverse virtual speakers. Additionally, while BWE helps bridge the frequency gap, other domain differences (like specific acoustic characteristics of telephone handsets) still pose challenges for future research.

Summary updated

Abstract

This paper investigates a novel data augmentation approach to train deep neural networks (DNNs) used for speaker embedding, i.e. to extract representation that allows easy comparison between speaker voices with a simple geometric operation. Data augmentation is used to create new examples from an existing training set, thereby increasing the quantity of training data improves the robustness of the model. We attempt to increase the number of speakers in the training set by generating new speakers via voice conversion. This speaker augmentation expands the coverage of speakers in the embedding space in contrast to conventional audio augmentation methods which focus on within-speaker variability. With an increased number of speakers in the training set, the DNN is trained to produce a better speaker-discriminative embedding. We also advocate using bandwidth extension to augment narrowband speech for a wideband application. Text-independent speaker recognition experiments in Speakers in the Wild (SITW) demonstrate a 17.9% reduction in minimum detection cost with speaker augmentation. The combined use of the two techniques provides further improvement.
BibTeX
@inproceedings{yamamoto2019speaker,
  author = {Hitoshi Yamamoto and Kong Aik Lee and Koji Okabe and Takafumi Koshinaka},
  title = {{Speaker Augmentation and Bandwidth Extension for Deep Speaker Embedding}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=dyjXDO-xNK},
}
Figure