A Weight Moving Average Based Alternate Decoupled Learning Algorithm for Long-Tailed Language Identification

Hui Wang, Lin Liu, Yan Song, Lei Fang, Ian McLoughlin, Li-Rong Dai2021 · DOI

TL;DR

This paper tackles the challenge of language identification (LID) in real-world scenarios where some languages are vastly more common than others (long-tailed data). They introduce WADCL, a novel algorithm that combines decoupled learning with a weight moving average technique to significantly improve LID performance on such imbalanced datasets.

What Problem Does This Paper Address?

Language identification (LID) systems are crucial for many applications, like multilingual speech recognition. While deep learning has greatly advanced LID, real-world data is often "long-tailed," meaning a few languages have abundant data, while many others have very few examples. This imbalance causes standard LID systems to perform poorly on the rare languages, as training is dominated by the majority classes. Existing solutions like re-sampling can either discard valuable data or lead to overfitting. This paper aims to bridge this gap by developing a robust LID system that excels even with highly imbalanced language data.

Methods

The proposed solution, WADCL (Weight Moving Average based Alternate Decoupled Learning), is built upon two core ideas:

  • Decoupled Learning (DCL): Instead of training a single end-to-end network, WADCL separates the LID system into two components: a feature extractor (frontend) and a classifier (backend). These are trained sequentially. The feature extractor is trained using random sampling to learn general patterns from the original imbalanced data. Then, the classifier is trained using balanced sampling to better learn decision boundaries for all classes, especially the minority ones. This decoupling helps to isolate the effects of representation learning from classifier adaptation.
  • Weight Moving Average (WMA): To further mitigate the negative side-effects of re-sampling (like information loss or overfitting), WADCL incorporates a WMA technique. Inspired by methods that average model parameters along the optimization trajectory, WMA fuses model weights from different stages of training. This acts as a form of implicit ensembling in the weight space, effectively averaging out the biases introduced by individual mini-batches and leading to better generalization, particularly for minority classes.

WADCL combines these by alternately training the feature extractor and a specific classifier branch using random sampling, and then training another classifier branch using balanced sampling, all while applying WMA to the learned weights.

Datasets

The authors created a new 6-language long-tailed dataset for their experiments. This dataset was constructed to mimic real-world applications, following an exponential distribution for language data. The imbalance factor (ratio of most frequent to least frequent class samples) was set to 40. The dataset was split into a long-tailed training set and a balanced test set, evaluated under 3s, 10s, and 30s duration conditions.

Key Insights and Ideas

  • Long-tailed data is a significant challenge for LID: Standard deep learning models struggle when language data is highly imbalanced, leading to poor performance on rare languages.
  • Decoupling helps isolate learning: Separating feature extraction and classification allows for tailored training strategies (random vs. balanced sampling) for each component, improving overall performance.
  • Weight averaging is key for generalization: WMA effectively mitigates the negative impacts of re-sampling by creating an ensemble of models in the weight space, leading to better performance on minority classes without sacrificing majority class performance.
  • Combining DCL and WMA is synergistic: The proposed WADCL algorithm leverages the strengths of both decoupled learning and weight moving average, achieving superior results compared to individual approaches.

Key Results

The proposed WADCL algorithm demonstrated significant performance improvements over state-of-the-art x-vector based LID methods on the long-tailed dataset.

  • For the 30s test condition, WADCL achieved an Equal Error Rate (EER) of 5.7%, compared to 7.3% for the x-vector baseline.
  • Across all test conditions (3s, 10s, 30s), WADCL consistently outperformed the x-vector system, with EERs of 18.4% (vs. 21.5%), 9.1% (vs. 11.6%), and 5.7% (vs. 7.3%) respectively.
  • Analysis of majority vs. minority class performance showed that WMA-based methods, particularly BS-WMA, significantly improved minority class accuracy while maintaining strong majority class performance, unlike simple re-sampling (BS) which often traded off performance.

Limitations and Future Work

  • Limitations: The paper notes that while WMA helps, it can still lead to higher training loss compared to standard methods. It also mentions that RS-WMA (random sampling with WMA) showed little benefit over RS, suggesting that random sampling might introduce biases that WMA alone cannot fully resolve.
  • Future Work: The authors plan to evaluate WADCL on other real-world imbalanced datasets, including the Arabic dialect recognition task from the 2019 multi-genre broadcast challenge, which features an even more extreme class imbalance (factor exceeding 50).

Summary updated

Abstract

<p dir="ltr">Language identification (LID) research has made tremendous progress in recent years, especially with the introduction of deep learning techniques. However, for real-world applications where the distribution of different language data is highly imbalanced, the performance of existing LID systems is still far from satisfactory. This raises the challenge of <i>long-tailed LID</i>. In this paper, we propose an effective weight moving average (WMA) based alternate decoupled learning algorithm, termed WADCL, for long-tailed LID. The system is divided into two components, a frontend feature extractor and a backend classifier. These are then alternately learned in an end-to-end manner using different sampling schemes to alleviate the distribution mismatch between training and test datasets. Furthermore, our WMA method aims to mitigate the side-effects of re-sampling schemes, by fusing the model parameters learned along the trajectory of stochastic gradient descent (SGD) optimization. To validate the effectiveness of the proposed WADCL algorithm, we evaluate and compare several systems over a language dataset constructed to match a long-tailed distribution based on real world application [1]. The experimental results from the long-tailed language dataset demonstrate that the proposed algorithm is able to achieve significant performance gains over existing state-of-the-art x-vector based LID methods.</p>
BibTeX
@misc{wang2021weight,
  author = {Hui Wang and Lin Liu and Yan Song and Lei Fang and Ian McLoughlin and Li-Rong Dai},
  title = {{A Weight Moving Average Based Alternate Decoupled Learning Algorithm for Long-Tailed Language Identification}},
  year = {2021},
  doi = {10.21437/interspeech.2021-776},
}
Figure