Language ID Prediction from Speech Using Self-Attentive Pooling

Roman Bedyakin, Nikolay MikhaylovskiyNAACL 2021

TL;DR

This paper presents a novel approach to Spoken Language Identification (LID) using a convolutional neural network with a Self-Attentive Pooling (SAP) layer. This method shows promising results, particularly for low-resource languages, by effectively focusing on the most informative parts of speech segments.

What Problem Does This Paper Address?

Spoken Language Identification (LID) is crucial for multilingual Automatic Speech Recognition (ASR) systems. However, identifying languages, especially low-resource or endangered ones, presents a significant challenge. These languages often have limited data, and existing systems struggle with speaker and domain variability. The paper aims to develop a robust LID system that can handle these challenges by intelligently processing speech signals.

Methods

The core of the proposed system is a convolutional neural network (CNN) architecture, specifically based on the QuartzNet ASR model. This model utilizes 1D Time-Channel Separable convolutions for its encoder. The key innovation lies in the Self-Attentive Pooling (SAP) layer used in the decoder. Instead of simply averaging frame-level features, SAP learns to assign different importance weights to different frames within an utterance. This is achieved by:

  1. Frame-level Feature Transformation: Each frame's feature map is passed through a fully-connected layer to obtain a hidden representation.
  2. Importance Weight Calculation: The similarity between each frame's hidden representation and a learnable context vector is computed. This similarity is then normalized using a softmax function to produce attention weights for each frame.
  3. Weighted Summation: The final utterance-level representation is generated as a weighted sum of the frame-level feature maps, where the weights are determined by the attention mechanism.

This self-attentive pooling allows the model to focus on the most discriminative parts of the speech signal for language identification, rather than being influenced equally by all segments. The system uses a standard cross-entropy loss function.

Datasets

  • Training Data: CMU Wilderness Dataset, containing read speech from 699 languages, typically with single-speaker recordings. The paper specifically mentions using derived MFCCs from this dataset.
  • Evaluation Data: A combination of datasets, including the Common Voice project and several OpenSLR corpora (SLR24, SLR35, SLR36, SLR64, SLR66, SLR79), as well as the Paradisec collection.
  • Specifics for the Task: The paper states that for the LID task, there are 16 languages in the released training data (4000 utterances per language). The validation and test data consist of 8000 utterances each, with 500 utterances per language.

Key Insights and Ideas

  • Self-Attentive Pooling for LID: The core idea that not all speech frames contribute equally to language identification is powerful. SAP allows the model to dynamically learn which parts of the speech signal are most indicative of a language, leading to improved performance, especially in challenging scenarios.
  • Effectiveness in Low-Resource Settings: The paper highlights promising results for low-resource settings, suggesting that the attention mechanism can help extract more information from limited data.
  • SpecAugment's Mixed Impact: Surprisingly, the authors found that SpecAugment (a data augmentation technique) did not consistently improve performance across all languages. It seemed to benefit Indo-European and Austronesian languages more than others, suggesting that its effectiveness might be language-family dependent. This opens avenues for further research into tailored augmentation strategies.
  • Potential Speaker Bias: The confusion matrix analysis suggests that gender of the speaker might be a confounding factor, leading to misclassifications. This is an important consideration for real-world LID systems.

Key Results

  • The proposed CNN with Self-Attentive Pooling achieved promising results on the validation and test sets.
  • Validation Set: While specific overall accuracy figures are not explicitly stated as a single number, the tables show F1-scores for individual languages. For example, without augmentation, "eng" (English) achieved an F1-score of 0.2738, and "tam" (Tamil) achieved 0.3074. With augmentation, "tam" improved to 0.3385.
  • Test Set: Similar to the validation set, individual language performance is reported. For instance, "tam" achieved an F1-score of 0.3191 without augmentation and 0.3385 with augmentation. The average F1-score across all languages was 0.1170 without augmentation and 0.1119 with augmentation, indicating that augmentation did not universally improve performance on the test set.
  • The paper notes that "iba" (Iban) was not detected at all without augmentation on the validation set, but showed some improvement with augmentation.

Limitations and Future Work

  • SpecAugment Dependency: The inconsistent performance of SpecAugment across different language families suggests that its application might need to be more nuanced or that alternative augmentation strategies could be explored.
  • Speaker Bias: The potential influence of speaker gender on LID performance is a limitation that warrants further investigation and mitigation strategies.
  • Further Research: The authors hypothesize that SpecAugment is more favorable for Indo-European and Austronesian languages, which requires further research to confirm and understand. They also suggest exploring other augmentation techniques.

Summary updated

BibTeX
@inproceedings{bedyakin2021language,
  author = {Roman Bedyakin and Nikolay Mikhaylovskiy},
  title = {{Language ID Prediction from Speech Using Self-Attentive Pooling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure