Recognition of Latin American Spanish Using Multi-Task Learning

Carlos Mendes, Alberto Abad, João Paulo Neto, Isabel TrancosoInterspeech 2019

TL;DR

This paper tackles the challenge of improving speech recognition for diverse Latin American Spanish accents by leveraging multi-task learning. They propose a novel method to automatically weight the contribution of different languages during training, showing significant improvements by incorporating European languages, particularly those with historical linguistic influence on Argentinian Spanish.

What Problem Does This Paper Address?

Speech recognition systems often struggle with accents, a problem that is amplified when dealing with the vast diversity of accents within a single language, especially in broadcast news where multiple speakers are present. This paper focuses on Latin American (LATAM) Spanish, a language with significant regional variations. The authors highlight that LATAM Spanish, particularly Argentinian Spanish, has been historically influenced by non-Spanish European migrations (like Italian and Portuguese). This linguistic cross-pollination presents a unique opportunity: can speech recognition performance for LATAM Spanish be improved by training models that also incorporate data from these influential European languages? Existing work on accented speech recognition has explored multi-task learning, but the authors aim to specifically investigate the impact of cross-language influence and propose a data-driven method for weighting these influences.

Methods

The core of the paper's methodology is multi-task learning (MTL) applied to deep neural networks (DNNs) for acoustic modeling. In MTL, a single model is trained to perform multiple related tasks simultaneously. Here, each language or accent is treated as a separate task. The DNN architecture consists of shared layers that learn general speech features and task-specific output layers for each language/accent.

A key innovation is their automatic per-language weight selection strategy. Instead of manually tuning weights for each task (language/accent), they propose a data-driven approach. This strategy leverages i-vectors, which are compact representations of speech segments capturing speaker and language characteristics. They compute an average i-vector for each language/accent and then use the cosine similarity between these i-vectors to determine the similarity between languages. This similarity score is then used to scale the cross-entropy loss for each language during training. The intuition is that languages more similar to the target (Argentinian Spanish) should have a higher weight in the training process, while dissimilar languages should have their contribution minimized.

Datasets

The paper uses broadcast news corpora for training and evaluation.

  • LATAM Spanish Accents: Approximately 30 hours of data per accent were used from the VoiceInteraction in-house broadcast news corpora. This includes Argentinian Spanish (es_AR) and other LATAM accents like Chilean (es_CL), Colombian (es_CO), Mexican (es_MX), and Peruvian (es_PE).
  • European Languages: Approximately 130 hours of data per language were used from the SAVAS project corpora. This includes European Spanish (es_ES), Italian (it_IT), and German (de_DE).

The data for each language/accent was split into training (0.8), development (0.1), and test (0.1) sets.

Key Insights and Ideas

  • Cross-language influence is beneficial: Incorporating data from languages that have historically influenced the target accent (Argentinian Spanish) significantly improves speech recognition performance.
  • Dialect similarity matters: The proposed i-vector based similarity measure effectively captures linguistic relationships. European Spanish and Italian, being more linguistically similar to Argentinian Spanish, yield better results when included in training compared to German.
  • Automatic weighting is crucial: The data-driven weighting strategy, based on i-vector similarity, is more effective than uniform weighting, especially for less similar languages like German, leading to noticeable improvements.
  • MTL for accent adaptation: Multi-task learning is a powerful framework for adapting acoustic models to specific accents, especially when leveraging related linguistic data.

Key Results

  • The baseline acoustic model trained solely on Argentinian Spanish achieved a Word Error Rate (WER) of 13.84%.
  • Using uniform weighting with European Spanish and Italian data resulted in relative improvements of 7.56% and 6.37% respectively.
  • The proposed dialect similarity-based weighted loss strategy showed improvements, particularly for German, where it achieved a 3.44% relative improvement when used alone, compared to no improvement with uniform weighting.
  • The best result was achieved by combining LATAM and European languages (es_AR, es_ES, it_IT, de_DE) using the weighted strategy, yielding a relative improvement of 13.33% and a final WER of 11.99%.

Limitations and Future Work

  • Limited European languages: The study focused on a few European languages. Future work could explore a wider range of influential languages.
  • Dialect similarity metric: While effective, the i-vector cosine similarity is one way to measure dialect similarity. Further exploration of this metric and its relation to training loss weights is suggested.
  • Comparison with other MTL techniques: The authors plan to compare their weighting strategy with other multi-task learning approaches, such as adversarial loss.
  • Broader application: The authors suggest extending this study to other accented languages, like Brazilian Portuguese, to validate the generalizability of their approach.

Summary updated

BibTeX
@inproceedings{mendes2019recognition,
  author = {Carlos Mendes and Alberto Abad and João Paulo Neto and Isabel Trancoso},
  title = {{Recognition of Latin American Spanish Using Multi-Task Learning}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure