Anlirika: An LSTM–CNN Flow Twister for Spoken Language Identification

Andreas Scherbakov, Liam Whittle, Ritesh Kumar, Siddharth Singh, Matthew Coleman, Ekaterina VylomovaNAACL 2021

TL;DR

This paper introduces "Anlirika," a novel LSTM-CNN model for Spoken Language Identification (SLI) designed for robustness across diverse domains and speakers. Despite facing a challenging task with limited, single-domain training data, their approach achieved a respectable 29.9% accuracy on the test set, highlighting the ongoing difficulty of truly speaker-invariant SLI.

What Problem Does This Paper Address?

Spoken Language Identification (SLI) is crucial for many natural language processing tasks, especially for the vast majority of the world's languages that are primarily oral. The core challenge is building systems that can accurately identify a language from speech, even when the audio comes from different speakers, recording conditions, or domains than what the system was trained on. This is particularly difficult for under-resourced languages where diverse training data is scarce. The SIGTYP 2021 Shared Task on Robust SLI specifically aimed to address this generalization problem, moving beyond single-speaker, single-domain scenarios.

Methods

The "Anlirika" system employs a multi-layer neural network architecture combining dense, convolutional, and recurrent layers. The design is motivated by a specific processing pipeline:

  • Dense Layer: Acts as a pre-processing step to remove sound harmonics by multiplying spectral patterns with a square matrix.
  • 1D CNN: Identifies features related to spectral line shape by convolving across the input feature vector index (frequency).
  • LSTM Layers: Capture "local" temporal constructs. The paper explores using 1 to 7 such layers.
  • Final LSTM Layer: Reduces temporal patterns into a single vector representation.
  • Dense Output Layer: Classifies the input into one of the 16 target languages.

A key aspect of their implementation is a custom batching mechanism designed to handle variable-length sequences efficiently. This involves transferring hidden states between batches, allowing for backpropagation through time across samples that might not perfectly align within a batch.

Datasets

  • Training Data: Derived from the CMU Wilderness dataset (Black, 2019), which consists of speech utterances from the Bible. This data is characterized by a single domain and predominantly a single speaker per language.
  • Validation and Test Data: Collected from multiple corpora, including Common Voice (Ardila et al., 2019). These sets feature diverse recording conditions and multiple speakers per language, posing a significant generalization challenge.
  • Languages: The dataset comprises 16 typologically diverse languages from various families (Afro-Asiatic, Austronesian, Basque, Dravidian, Indo-European, Niger-Congo, and Tai-Kadai).

Key Insights and Ideas

  • The Challenge of Generalization: The paper strongly emphasizes that even with a constrained task (no external data or augmentation from validation sets), SLI remains challenging. The significant drop in accuracy from validation to test sets highlights the difficulty of achieving speaker-invariant representations.
  • Data Augmentation Impact: Augmenting the training data with validation set samples led to a dramatic improvement in accuracy (74% on validation), suggesting that speaker overlap between training and validation is a major factor in performance. This also led to a noticeable improvement in convergence dynamics.
  • Hyperparameter Sensitivity: The choice of the number of sequential LSTM layers ($N_L$) significantly impacted performance, with $N_L=2$ yielding the highest accuracy. Increasing the number of units in LSTM layers ($D_L$) from 200 to 300 did not yield significant improvements.
  • Batching for Efficiency: The custom batching mechanism, while potentially limiting temporal depth in backpropagation, is presented as a practical solution for handling variable-length sequences in LSTMs.

Key Results

  • Best Test Accuracy: The "Anlirika" system achieved 29.9% accuracy on the test set when trained on the augmented data (training + validation samples).
  • Validation Accuracy: On the validation set, the system achieved 43.6% accuracy (Micro Avg) and 42.1% accuracy (Macro Avg) when using the final model version.
  • Initial Training: Using only the original training set, the model struggled to learn, achieving only about 12% accuracy on the validation set and showing a poor correlation between train and validation accuracy figures.

Limitations and Future Work

  • Generalization Gap: The primary limitation identified is the significant gap between performance on validation and test sets, indicating that the system still struggles to generalize to unseen speakers and domains.
  • Batching Drawback: The batching mechanism, while efficient, might constrain the temporal depth of backpropagation through time in LSTM layers, potentially affecting learning. The authors note this effect as minor but suggest it warrants further investigation.
  • Future Work: The authors suggest exploring architecture modifications that directly enforce phonetic generalization, such as incorporating "bottlenecks" (layers with low output size), to improve robustness. They also plan to continue their experiments to further understand the model's behavior.

Summary updated

BibTeX
@inproceedings{scherbakov2021anlirika,
  author = {Andreas Scherbakov and Liam Whittle and Ritesh Kumar and Siddharth Singh and Matthew Coleman and Ekaterina Vylomova},
  title = {{Anlirika: An LSTM–CNN Flow Twister for Spoken Language Identification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure