NAIST’s Machine Translation Systems for IWSLT 2020 Conversational Speech Translation Task

Ryo Fukuda, Katsuhito Sudoh, Satoshi NakamuraACL 2020

TL;DR

This paper tackles the challenge of translating disfluent conversational speech transcripts, often riddled with Automatic Speech Recognition (ASR) errors, into fluent English. The authors propose a novel domain adaptation method that uses unsupervised style transfer to make out-of-domain data more similar to the in-domain data, leading to improved translation accuracy.

What Problem Does This Paper Address?

Machine Translation (MT) has seen significant advancements, but translating domain-specific and noisy text remains a challenge, especially in low-resource scenarios. Conversational speech, in particular, presents unique difficulties due to disfluencies, filler words, and ASR errors. The IWSLT 2020 Conversational Speech Translation Task specifically focuses on translating disfluent Spanish speech transcripts into fluent English. Standard domain adaptation techniques, which often rely on large amounts of parallel data, struggle when the out-of-domain data's style (e.g., fluent written text) is very different from the in-domain data (disfluent speech transcripts). This paper aims to bridge this gap by developing a method to effectively leverage out-of-domain data even when its similarity to the target domain is low.

Methods

The proposed method has two main components:

  1. Unsupervised Style Transfer: To address the lack of parallel data for fluent-to-disfluent Spanish translation, the authors employ an unsupervised neural machine translation (UNMT) approach. This technique, based on Lample et al. (2018a,b), learns to translate between languages using only monolingual corpora. Here, it's adapted to transfer the style of out-of-domain fluent Spanish data to mimic the disfluent style of the in-domain Fisher data. This creates "pseudo-in-domain" parallel data.
  2. Domain Adaptation (Fine-tuning): A standard fine-tuning approach is used for the main translation task. An NMT model is first pre-trained on a large, resource-rich out-of-domain dataset (or the pseudo-in-domain data generated in step 1). Subsequently, this pre-trained model is fine-tuned on the smaller, low-resource in-domain dataset. This process allows the model to adapt its knowledge from the general domain to the specific characteristics of the target task.

Datasets

  • In-domain data: LDC Fisher Spanish speech (disfluent) with English translations (fluent). The task organizers provided several versions, including ASR output transcripts and fluent English translations. For training, the ASR output (Spanish disfluent) was used as input, and fluent English translations were used as output.
  • Out-of-domain data: United Nations Parallel Corpus (UN-Corpus). This corpus consists of manually translated UN documents across six languages, including Spanish and English. For this study, one million Spanish-English bilingual sentence pairs were randomly selected.

Key Insights and Ideas

  • Style Transfer for Domain Adaptation: The core idea is that increasing the stylistic similarity between out-of-domain and in-domain data can significantly improve domain adaptation performance, even when the content is different.
  • Unsupervised Style Transfer is Crucial: The lack of parallel data for fluent-to-disfluent Spanish necessitates unsupervised methods for style transfer.
  • Pseudo-in-domain Data Benefits: Generating pseudo-in-domain data through style transfer and then using it for pre-training or as part of the training data for the main NMT model leads to better results than using only the original out-of-domain data.
  • Fluent References Improve Performance: Using fluent English translations as references for training the NMT model yielded better BLEU scores compared to using disfluent references, suggesting that the target output should be clean.

Key Results

  • The proposed domain adaptation method, incorporating style transfer, outperformed a baseline model that did not use style transfer.
  • Specifically, models trained using the "Fisher-like UNCorpus" (UN-Corpus data style-transferred to be disfluent) combined with Fisher data achieved BLEU scores of 18.5 (Table 3) and 15.6 (Table 4), which were higher than models trained solely on Fisher (14.8 and 11.6) or UNCorpus (7.8 and 15.2).
  • The perplexity of the out-of-domain data (UNCorpus) when measured against a language model trained on in-domain data (Fisher) was significantly reduced by style transfer (from 589.81 to 474.47), indicating increased domain similarity.
  • The study found that removing disfluency from reference sentences improved BLEU scores by approximately three points.

Limitations and Future Work

  • Data Quality Degradation: A limitation of the style transfer method was that it sometimes led to the loss of sentence meaning, degrading the quality of the parallel data and consequently the translation performance. The authors suggest that future work should focus on models that can control the trade-off between style transfer and content preservation.
  • In-domain Disfluent References: The loss of accuracy caused by not using clean (fluent) reference sentences for the in-domain data could not be fully resolved by domain adaptation alone.
  • Future Work: The authors plan to pursue style transfer systems that preserve sentence meaning while converting style. This could lead to improved translation accuracy for noisy input sentences and the generation of fluent outputs by removing output disfluencies.

Summary updated

BibTeX
@inproceedings{fukuda2020naists,
  author = {Ryo Fukuda and Katsuhito Sudoh and Satoshi Nakamura},
  title = {{NAIST’s Machine Translation Systems for IWSLT 2020 Conversational Speech Translation Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure