TL;DR
This paper introduces TALL, a novel approach to Cross-Lingual Transfer Learning (CLTL) for Natural Language Understanding (NLU) that integrates Unsupervised Machine Translation (UMT). By pre-training an NLU model with a machine translation objective using UMT, TALL significantly improves performance, especially for distant language pairs, without requiring additional annotated data.
What Problem Does This Paper Address?
Virtual assistants rely heavily on Natural Language Understanding (NLU) to interpret user commands. Training NLU models typically requires large amounts of labeled data, which is readily available for high-resource languages but scarce for low-resource languages. This "data hungriness" limits the effectiveness of NLU models when deployed in less common languages. Cross-Lingual Transfer Learning (CLTL) aims to bridge this gap by leveraging data from high-resource languages to train models for low-resource ones. However, existing CLTL methods, even those using powerful pre-trained multilingual language models (like M-BERT), struggle with distant language pairs due to limited correlation in their learned representations. This paper addresses the challenge of improving CLTL, particularly for distant languages, by enhancing the cross-lingual representation space.
Methods
The core innovation is the Translation Aided Language Learner (TALL) model. TALL builds upon a pre-trained multilingual language model (acting as the encoder) and augments it with a decoder to form a transformer-based sequence-to-sequence architecture. The key is a two-stage training process:
- MT-Oriented Pre-training: TALL is first pre-trained on a machine translation (MT) task between the source and target languages. Crucially, this pre-training utilizes Unsupervised Machine Translation (UMT) techniques. UMT allows the model to learn translation without parallel (aligned) corpora, relying instead on large amounts of monolingual data. This is achieved through two main UMT losses:
- Denoising Auto-Encoding Loss: The model learns to reconstruct sentences after noise (token dropping/swapping) has been introduced.
-
Back-Translation Loss: The model learns to translate a sentence to the target language and then back to the source language, aiming to recover the original sentence. This MT pre-training is designed to explicitly enhance the correlation between the source and target languages in the model's representation space.
-
NLU-Oriented Fine-tuning: After MT pre-training, TALL is fine-tuned as an NLU model for slot filling and intent classification, similar to how the baseline model is trained. The authors adapt the fine-tuning strategy from BART, using both the encoder and decoder for NLU prediction.
The baseline model for comparison is simply the pre-trained multilingual language model (encoder) directly fine-tuned for NLU tasks.
Datasets
- For UMT Pre-training:
- Wikipedia dumps were used to collect large amounts of unannotated monolingual sentences for both source and target languages.
- For CLTL Evaluation:
- MultiATIS++: A multilingual NLU dataset derived from the ATIS dataset, focusing on airline travel, with 5K annotated sentences per language.
- Multi-domain NLU Dataset: A larger, multi-domain dataset (100K sentences per language) collected from a virtual assistant, covering domains like music, notifications, smart home, weather, and books.
Key Insights and Ideas
- UMT as a Bridge for Distant Languages: The paper demonstrates that leveraging UMT for MT pre-training is a powerful way to improve CLTL, especially for language pairs that are linguistically distant. This suggests that the MT objective effectively forces the model to learn deeper cross-lingual alignments that are not naturally captured by standard multilingual pre-training.
- MT Pre-training Enhances Representation Space: The MT-oriented pre-training explicitly aims to create a shared representation space where source and target languages are more correlated. This enhanced correlation directly benefits the subsequent NLU fine-tuning.
- No Additional Annotated Data Needed: A significant advantage of TALL is that it achieves better CLTL performance without requiring any extra labeled data for the NLU task itself, relying solely on the readily available unannotated corpora for UMT.
- Decoder's Importance for CLTL: Ablation studies show that the decoder, introduced for the MT pre-training, is crucial for TALL's strong CLTL performance during fine-tuning, highlighting the benefit of the sequence-to-sequence architecture.
Key Results
- Consistent CLTL Improvement: TALL consistently outperformed the baseline pre-trained multilingual language model across various CLTL tasks and language pairs.
- Prominent Gains for Distant Languages: The performance gain of TALL over the baseline was particularly significant for distant language pairs (e.g., English-Japanese, German-Japanese), demonstrating its effectiveness in challenging scenarios. For instance, on the MultiATIS++ dataset, TALL achieved gains of up to 36.75% in Slot F1 for distant languages.
- Competitive Performance on Similar Languages: While the gains were more pronounced for distant languages, TALL also showed improvements for similar language pairs (e.g., English-German).
- Translation Performance: The MT-oriented pre-training of TALL also yielded good translation performance, as measured by BLEU scores on Wikipedia data, indicating the effectiveness of the UMT approach.
Limitations and Future Work
- Startup Supervision for Back-Translation: The authors explored using bilingual dictionaries for "startup supervision" in back-translation but found no performance gain, suggesting that the pre-trained encoder already provides a sufficiently initialized representation space.
- UMT Training's Impact: The UMT training, through denoising auto-encoding and back-translation, enhances the correlation between language pairs. The paper posits that this enhancement is more impactful for distant languages than similar ones, leading to larger relative gains.
- Future Work: The authors suggest collecting unannotated corpora that are linguistically compatible with specific downstream NLU tasks for UMT training, believing this could further boost TALL's CLTL performance.
Summary updated