Pair-Wise Distance Metric Learning of Neural Network Model for Spoken Language Identification

Xugang Lu, Peng Shen, Yu Tsao, Hisashi Kawai2016 · DOI

TL;DR

This paper introduces a novel approach to spoken language identification (SLI) by integrating pair-wise distance metric learning into deep neural networks (DNNs). By explicitly training the DNN to minimize the distance between i-vectors from the same language and maximize the distance between i-vectors from different languages, the model achieves a significant 20% relative improvement in identification error rate, addressing the common issue of overfitting in DNNs with limited training data.

What Problem Does This Paper Address?

Spoken Language Identification (SLI) aims to determine the language of an audio utterance. A popular technique for this is using i-vectors, which are fixed-length representations of speech that capture acoustic variations. However, i-vectors also encode irrelevant variations like speaker identity or channel noise. To make them more useful for SLI, discriminative transforms are needed to emphasize language-specific features. Deep Neural Networks (DNNs), with their powerful non-linear modeling capabilities, have shown promise for this.

The challenge arises when training DNNs for SLI, especially with limited data. Standard DNN training often uses point-wise feature-label information, optimizing based on the difference between predicted and true labels. This can lead to overfitting, where the model performs well on training data but poorly on unseen data. The authors aim to improve DNN generalization for SLI by incorporating more information from the training data, specifically the relationships between pairs of samples.

Methods

The core contribution is the integration of pair-wise distance metric learning into the DNN training framework for SLI. Instead of just learning a direct mapping from i-vectors to language labels (point-wise learning), the authors propose to explicitly constrain the DNN's hidden layers to learn a representation space where:

  • Intra-class variation is minimized: Samples from the same language are pushed closer together.
  • Inter-class variation is maximized: Samples from different languages are pulled further apart.

This is achieved by defining a pair-wise loss function based on the cosine distance between representations of paired i-vectors from the last hidden layer. This loss is then combined with the standard cross-entropy loss (point-wise learning) in a unified objective function. The DNN parameters are then optimized using stochastic gradient descent (SGD) to minimize this combined loss. The paper focuses on applying this metric learning constraint to the last hidden layer of the DNN.

Datasets

The experiments were conducted using a dataset from the NIST i-vector challenge for SLI. This dataset includes:

  • 50 languages.
  • Each utterance is represented as a 400-dimension i-vector.

The dataset was split into:

  • Training: 10,000 samples (200 per language), with all possible pair-wise combinations used for metric learning.
  • Validation: 2,500 samples.
  • Testing: 2,500 samples (50 per language).

Key Insights and Ideas

  • Pair-wise relationships are crucial for generalization: Explicitly modeling the relative distances between sample pairs (same vs. different languages) provides a stronger signal for learning than just point-wise labels, especially when data is scarce.
  • DNNs as metric learners: The hidden layers of a DNN can be viewed as learning a non-linear distance metric. By adding a specific metric learning objective, we can guide this learning process more effectively.
  • Balancing point-wise and pair-wise objectives: The proposed method successfully balances the standard classification objective (cross-entropy) with the new metric learning objective, allowing for improved performance without sacrificing classification accuracy.
  • Overfitting mitigation: The pair-wise metric learning acts as a form of regularization, preventing the DNN from memorizing the training data and thus improving its ability to generalize to unseen data.

Key Results

The proposed pair-wise distance metric learning approach demonstrated significant improvements:

  • A relative improvement of over 20% in the identification error rate was achieved on the SLI task.
  • For the M2*512 model (two hidden layers), the best performance was obtained with a regularization coefficient ($\gamma$) of 0.01, yielding a test error rate of 14.42%. This is a substantial improvement over the baseline M2*512 model's test error rate of 18.32%.
  • Even simpler models (M1*512) benefited, showing a relative improvement of 20.01%.
  • The performance of the proposed method was superior to a conventional linear modeling technique (LDA+SVM), which achieved a test error rate of 16.28%.

Limitations and Future Work

  • Focus on the last hidden layer: The current study primarily applied the pair-wise metric learning constraint to the last hidden layer. The authors suggest that applying it to all hidden layers could lead to further improvements.
  • Integration strategies: While the regularization framework (combining losses) showed good results, the authors also explored metric learning as a pre-training step followed by fine-tuning with a softmax layer. They note that this approach yielded improvement but was not as significant as the regularization framework. Further investigation into different integration strategies is suggested.
  • Dataset size: While the paper demonstrates effectiveness on a dataset with limited samples per language, the performance gains might vary with larger datasets.

The authors express optimism about the potential of pair-wise metric learning for DNNs in SLI and plan to explore more advanced integration techniques and architectures in future work.

Summary updated

BibTeX
@misc{lu2016pair,
  author = {Xugang Lu and Peng Shen and Yu Tsao and Hisashi Kawai},
  title = {{Pair-Wise Distance Metric Learning of Neural Network Model for Spoken Language Identification}},
  year = {2016},
  doi = {10.21437/interspeech.2016-722},
}
Figure