Linguistically-Informed Training of Acoustic Word Embeddings for Low-Resource Languages

Zixiaofan Yang, Julia HirschbergInterspeech 2019

TL;DR

This paper introduces novel methods for training acoustic word embeddings that better capture linguistic relationships like stemming and pronunciation, improving performance for low-resource languages. By incorporating these linguistic cues, the models can more effectively identify variations of keywords, a crucial capability for applications like spoken keyword search in languages with limited data.

What Problem Does This Paper Address?

The field of speech processing, particularly for low-resource languages, faces a significant challenge: building effective systems with limited data. A key application is query-by-example keyword search, where users can find spoken instances of a keyword. Traditional methods for training acoustic word embeddings, which represent spoken words as fixed-dimensional vectors, often rely on exact orthographic matches. This means that words with the same stem or similar pronunciation (e.g., "terrorist" and "terrorism") are treated as completely different, leading to poor performance when users search for variations of a keyword. This paper aims to bridge this gap by developing embedding training methods that leverage linguistic information to handle these non-exact matches, making them more suitable for real-world low-resource scenarios.

Methods

The core of the paper's contribution lies in two linguistically-informed training methods for acoustic word embeddings, built upon a Siamese neural network architecture with a triplet loss function.

  • Standard Siamese Network with Triplet Loss: This is the baseline approach. A Siamese network takes pairs of spoken word samples and learns to produce similar embeddings for samples from the same word class and dissimilar embeddings for samples from different word classes. The triplet loss function aims to maximize the distance between an "anchor" sample and a "negative" sample (from a different class) while minimizing the distance between the anchor and a "positive" sample (from the same class), with a margin m.
  • Linguistically-Informed Method 1: Clustering Words by Stems: This method addresses the issue of morphological variations. Instead of treating each unique word as a separate class, words sharing the same stem are grouped into a single class. The triplet loss is then applied, where a positive sample comes from the same stem group, and a negative sample comes from a different stem group. This encourages the model to focus on the core meaning of words rather than their exact spelling.
  • Linguistically-Informed Method 2: Learning Pronunciation Distance: This method tackles the problem of similar-sounding words. The standard triplet loss treats all negative samples equally. This new approach modifies the loss function to directly incorporate phonetic information. It aims to minimize the squared difference between the embedding distance of two words and their phonetic edit distance. This encourages embeddings of phonetically similar words to be closer, even if they are orthographically different.

The paper also details the neural network architecture, which includes recurrent layers (specifically Bidirectional LSTMs) for temporal context, an attention layer to focus on relevant frames, and fully connected layers to compute the final embeddings.

Datasets

The experiments were conducted on two main settings:

  • Low-Resource Setting (English Switchboard): A limited subset of the Switchboard corpus was used. This subset involved approximately 2 hours of training speech, simulating a low-resource scenario. The data was segmented into words based on existing word-aligned transcripts.
  • Zero-Resource Setting (Sinhala): For this setting, the models were trained on the full English Switchboard dataset and then tested on spoken Sinhala keywords. This simulates a scenario where no transcribed data is available in the target language. Sinhala keywords were collected from native speakers and recorded, resulting in 121 unique words and 610 spoken samples.

Key Insights and Ideas

  • Linguistic Information is Crucial for Low-Resource Keyword Search: The core insight is that traditional embedding training, relying on exact word matches, is insufficient for real-world applications like keyword search where variations are common. Incorporating linguistic knowledge (stems, pronunciation) significantly improves performance.
  • Stemming for Morphological Variations: Grouping words by their stems is an effective strategy for handling different forms of the same word (e.g., plurals, verb conjugations). This allows models to generalize better to unseen but related words.
  • Phonetic Similarity for Acoustic Retrieval: Directly learning phonetic distance helps in retrieving words that sound alike, which is important for spoken queries where pronunciation can vary or be imperfect.
  • Cross-Lingual Transfer is Feasible: The zero-resource experiments demonstrate that models trained on a high-resource language (English) can be effectively transferred to a low-resource language (Sinhala) for embedding spoken words, even without language-specific adaptation data.
  • Visualizing Embeddings Reveals Linguistic Structure: t-SNE visualizations of the Sinhala embeddings showed clear clusters of semantically and phonetically similar words, visually confirming the effectiveness of the proposed methods.

Key Results

  • Switchboard (Low-Resource):
    • The "Stem Triplet" model achieved a 6.3% higher Stem AP than the baseline "Word Triplet" model, indicating better performance in identifying words with morphological variations, while only sacrificing 2.2% in Word AP (exact match accuracy).
  • The "Pronunciation Dist" model significantly outperformed others in Phonetic Sim (38.8% vs. 23.3% and 21.7%), demonstrating its ability to capture phonetic similarity.
  • Sinhala (Zero-Resource):
    • The "Stem Triplet" model trained on English achieved the highest Word AP (60.3%) when tested on Sinhala spoken words, outperforming the baseline "Word Triplet" (57.2%).
  • All models achieved high Precision@4 (around 80%), indicating good retrieval of other samples of the same word.
  • t-SNE visualizations showed distinct clusters for semantically related Sinhala words, such as {destroy, destroyed, destruction} and {attack, attacker}.

Limitations and Future Work

  • Limitations:
    • The "Stem Triplet" model slightly sacrifices performance on exact word matching compared to the baseline.
  • In the zero-resource Sinhala setting, an accurate stemmer or pronunciation dictionary was not available, limiting the evaluation metrics.
  • The paper notes that their best AP on a similar Switchboard setup was 67.1%, but this used significantly more data for normalization than their low-resource scenario.
  • Future Work:
    • Explore combining the proposed training methods to achieve embeddings that excel in both word clustering and distance measurement.
  • Conduct query-by-example keyword search experiments in low-resource and zero-resource settings using the developed methods.

Summary updated

BibTeX
@inproceedings{yang2019linguistically,
  author = {Zixiaofan Yang and Julia Hirschberg},
  title = {{Linguistically-Informed Training of Acoustic Word Embeddings for Low-Resource Languages}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure