Self Multi-Head Attention for Speaker Recognition

Miquel India, Pooyan Safari, Javier HernandoInterspeech 2019

TL;DR

This paper introduces a novel "self multi-head attention" pooling mechanism for speaker recognition. By allowing a neural network to selectively focus on different parts of a speech signal, this method significantly improves the accuracy of identifying speakers, outperforming existing state-of-the-art techniques.

What Problem Does This Paper Address?

Speaker recognition systems aim to identify individuals based on their voice. Traditional approaches often rely on extracting a single, fixed-length vector (an embedding) that represents a speaker's unique vocal characteristics. Many current deep learning methods achieve this by processing short segments of speech, extracting embeddings from each, and then averaging them to get an utterance-level representation. However, this averaging approach can dilute discriminative information, especially for short speech samples. The challenge is to create a robust speaker embedding that effectively captures the most important vocal cues from variable-length utterances, even when those cues are spread across different parts of the speech signal.

Methods

The core innovation in this paper is the Self Multi-Head Attention Pooling mechanism. This builds upon the concept of attention, where a model learns to assign different weights to different parts of an input sequence.

Here's how it works:

  1. CNN Encoder: A Convolutional Neural Network (CNN) first processes the input speech spectrograms to extract short-term acoustic features, creating a sequence of feature vectors.
  2. Self Attention: Instead of simply averaging these feature vectors, the self-attention mechanism learns to assign a "relevance" weight to each vector. This is done using a trainable parameter that helps the model decide which parts of the sequence are most important for speaker identification.
  3. Multi-Head Attention: This is the key enhancement. The attention mechanism is applied multiple times in parallel ("multiple heads"). Each "head" learns to focus on different aspects or sub-spaces of the encoded features. This allows the model to capture diverse discriminative patterns from various segments of the speech utterance simultaneously.
  4. Pooling: The weighted features from each head are then combined to form a final utterance-level speaker embedding.

This multi-head approach differs from simpler attention mechanisms by allowing the model to attend to different "views" of the data, rather than a single global view.

Datasets

  • VoxCeleb1: This is the primary dataset used for both training and evaluation. It's a large-scale, in-the-wild speech dataset containing utterances from celebrities. The authors used the standard VoxCeleb1 speaker verification protocol, training on the development partition and evaluating on the test set.

Key Insights and Ideas

  • Beyond Averaging: Simple averaging of speaker embeddings from short segments is a bottleneck. Attention mechanisms offer a more sophisticated way to pool information.
  • The Power of Multiple Perspectives: The "multi-head" aspect is crucial. It allows the model to learn different, complementary ways to represent a speaker by focusing on distinct feature subsets and temporal regions. This is like having multiple experts analyze the speech, each with a different specialization.
  • Focusing on Discriminative Features: The attention weights learned by the model highlight specific segments of the speech that are most informative for distinguishing speakers. This is more effective than treating all segments equally.
  • Efficiency: The multi-head attention mechanism achieves improved performance without significantly increasing the model's complexity or parameter count compared to single-head attention.

Key Results

The proposed Self Multi-Head Attention (MHA) pooling significantly outperforms baseline methods on the VoxCeleb1 speaker verification task:

  • Error Rate Reduction: The MHA pooling achieved an Equal Error Rate (EER) of 4.0%.
  • Comparison to Baselines:
    • i-vector + PLDA: 9.54% EER (a 58% relative improvement in EER for MHA).
  • CNN + Temporal Pooling: 4.91% EER (an 18% relative improvement for MHA).
  • CNN + Statistical Pooling: 4.9% EER (an 18% relative improvement for MHA).
  • CNN + Vanilla Attention Pooling: 4.71% EER (a 14% relative improvement for MHA).

The DET curves (Figure 3) visually confirm that MHA pooling provides superior performance across various operating points.

Limitations and Future Work

  • Limitations: The paper doesn't explicitly detail limitations of the MHA approach itself, but the performance gains are relative to existing methods. The effectiveness might still depend on the quality and length of the input speech.
  • Future Work: Not explicitly discussed in the provided text, but the success of multi-head attention suggests further exploration in different attention architectures and applications within speech processing. The authors mention that the network is trained as a speaker classifier, and the bottleneck layer before the softmax is used as the speaker embedding, implying that end-to-end training for verification could be a future direction.

Summary updated

BibTeX
@inproceedings{india2019self,
  author = {Miquel India and Pooyan Safari and Javier Hernando},
  title = {{Self Multi-Head Attention for Speaker Recognition}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure