Simple Attention Module Based Speaker Verification with Iterative Noisy Label Detection

Dan Su, Xiaoyi Qin, Na Li, Chao Weng, Dan Su 0002, Ming Li

TL;DR

This paper introduces SimAM, a novel, parameter-free attention module for speaker verification that improves model performance by focusing on important features. They also propose an iterative noisy label detection method to clean training data, significantly reducing errors caused by mislabeled samples.

What Problem Does This Paper Address?

Automatic Speaker Verification (ASV) systems, which aim to identify who is speaking, have seen significant improvements with deep learning. However, two key challenges remain: 1) effectively capturing relevant features within speech signals, and 2) dealing with the pervasive issue of noisy labels in large-scale, human-annotated datasets. Existing attention mechanisms like SE and CBAM have shown promise, but the authors argue they might not be optimal for the temporal nature of speech. Furthermore, noisy labels can lead deep neural networks to "memorize" incorrect information, degrading performance. This paper tackles both these issues by proposing a new attention module and a data cleaning strategy.

Methods

The core contributions are two-fold:

  • Simple Attention Module (SimAM): This is a "plug-and-play" module designed to be easily integrated into existing deep learning architectures for ASV. Unlike some other attention mechanisms, SimAM doesn't require extra learnable parameters. It works by defining an energy function for each neuron in a feature map, aiming to minimize the difference between the target neuron's output and other neurons' outputs. This process effectively generates 3D attention weights (across channel, frequency, and time dimensions) that highlight the most important features for speaker verification. The paper notes that this 3D attention is particularly well-suited for speech signals, which are inherently temporal.
  • Iterative Noisy Label Detection: To address the problem of mislabeled data, the authors propose an iterative approach. The core idea is to leverage speaker embeddings. For each training utterance, they calculate its cosine similarity with the average embedding of its supposed speaker. Utterances with consistently low similarity across multiple iterations are flagged as potentially noisy and removed from the training set. This process is repeated until no significant improvements are observed, effectively cleaning the dataset.

Datasets

  • Training: VoxCeleb 2 development set (5,994 speakers, 1,092,009 utterances). Data augmentation techniques like adding noise, reverberation, and speed perturbation were applied.
  • Evaluation: VoxCeleb 1 dataset, specifically three trial lists:
  • VoxCeleb 1-0 (original)
    • VoxCeleb 1-E (extended)
    • VoxCeleb 1-H (hard)

Key Insights and Ideas

  • 3D Attention for Speech: The paper highlights that speech signals have a strong temporal component. SimAM's ability to generate 3D attention weights, considering channel, frequency, and time, is argued to be more effective than 2D attention mechanisms for capturing these temporal dynamics in ASV.
  • Parameter-Free Attention: The fact that SimAM is parameter-free is a significant advantage, as it can be added to existing models without increasing their complexity or requiring extensive retraining of the new parameters.
  • Iterative Data Cleaning: The iterative nature of the noisy label detection is crucial. It allows the system to refine its understanding of what constitutes a "noisy" utterance as the speaker embeddings become more accurate through training. This is a more robust approach than a single-pass filtering method.
  • Memorization Effect: The paper correctly identifies that noisy labels can lead DNNs to overfit and memorize incorrect associations, which is a well-known problem in supervised learning. Their noisy label detection directly combats this.

Key Results

  • SimAM Performance: When integrated into a ResNet34 backbone, SimAM achieved an Equal Error Rate (EER) of 0.675% on the VoxCeleb 1 original test trials. This represents a significant improvement over baseline models and even surpasses the performance of SE-ResNet34 by 5% relative improvement.
  • Noisy Label Detection Impact: The iterative noisy label detection method further reduced the EER to 0.643%, demonstrating an additional 7% relative improvement over the system with SimAM alone.
  • Model Size: SimAM adds minimal overhead in terms of model size compared to the base ResNet34 architecture.

Limitations and Future Work

  • Noisy Label Thresholding: The effectiveness of the noisy label detection method relies on setting an appropriate cosine similarity threshold. The paper mentions that this threshold might need to be adjusted based on the dataset and specific task.
  • Generalizability of SimAM: While shown to be effective on VoxCeleb, further research could explore SimAM's performance on other speech-related tasks or different datasets.
  • Automated Threshold Selection: The paper doesn't explicitly discuss methods for automatically determining the optimal threshold for noisy label detection, which could be a direction for future work.
  • Computational Cost of Iterative Cleaning: While the method is effective, the iterative nature might increase the overall training time. Investigating more efficient iterative schemes could be beneficial.

Summary updated

Abstract

Recently, the attention mechanism such as squeeze-and-excitation module (SE) and convolutional block attention module (CBAM) has achieved great success in deep learning-based speaker verification system. This paper introduces an alternative effective yet simple one, i.e., simple attention module (SimAM), for speaker verification. The SimAM module is a plug-and-play module without extra modal parameters. In addition, we propose a noisy label detection method to iteratively filter out the data samples with a noisy label from the training data, considering that a large-scale dataset labeled with human annotation or other automated processes may contain noisy labels. Data with the noisy label may over parameterize a deep neural network (DNN) and result in a performance drop due to the memorization effect of the DNN. Experiments are conducted on VoxCeleb dataset. The speaker verification model with SimAM achieves the 0.675% equal error rate (EER) on VoxCeleb1 original test trials. Our proposed iterative noisy label detection method further reduces the EER to 0.643%.
BibTeX
@inproceedings{su2022simple,
  author = {Dan Su and Xiaoyi Qin and Na Li and Chao Weng and Dan Su and Ming Li},
  title = {{Simple Attention Module Based Speaker Verification with Iterative Noisy Label Detection}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746294},
}
Figure