On the cross-lingual transferability of multilingual prototypical models across NLU tasks

Oralie Cattan, Sophie Rosset, Christophe ServanACL 2021

TL;DR

This paper tackles the challenge of building Natural Language Understanding (NLU) systems for languages with limited data. They propose a novel approach combining few-shot learning with multilingual Transformer models (like mBERT) and prototypical networks, demonstrating significant improvements in cross-lingual transferability, especially for under-resourced languages.

What Problem Does This Paper Address?

Natural Language Understanding (NLU) systems are crucial for dialogue systems, enabling them to extract meaning from user queries. Traditionally, NLU relies on large amounts of labeled data, which is readily available for high-resource languages like English. However, this poses a significant hurdle for under-resourced languages, where obtaining such data is prohibitively expensive and time-consuming. Existing approaches often struggle with domain-specific designs and the inherent language-dependency of NLU tasks, making them difficult to adapt to new languages. While multilingual models like BERT have shown promise in cross-lingual transfer, they still exhibit limitations when applied to languages with very little data. This paper aims to bridge this gap by investigating how to effectively transfer NLU capabilities to low-resource languages using limited data.

Methods

The core of this work lies in the synergistic combination of two key techniques:

  • Prototypical Neural Networks (PNNs): Inspired by human learning, PNNs learn to classify new examples by comparing them to "prototypes" representing each class. These prototypes are essentially centroids of embedded data points belonging to a class. The paper leverages PNNs for their effectiveness in few-shot learning, where models can learn from very few examples.
  • Multilingual Transformers (mBERT): The authors utilize a pre-trained multilingual BERT (mBERT) model as the encoder. mBERT is trained on a massive corpus of text from many languages, allowing it to learn rich, cross-lingually transferable representations.

The proposed approach involves fine-tuning the mBERT encoder and then using it within a PNN framework. This allows the model to learn a shared embedding space where class prototypes can be formed. The training is conducted using an "episodic learning" strategy, where the model is presented with a series of "episodes," each mimicking a few-shot learning scenario (N-way, k-shot classification). This episodic training helps the model generalize to unseen classes and languages. The paper also explores both "target only" (training only on the target language's limited data) and "multilingual" (training on a combination of all languages) configurations, as well as a "multilingual zero-shot" scenario.

Datasets

The primary dataset used in this research is the MultiATIS++ corpus. This corpus is a multilingual extension of the ATIS (Air Travel Information System) corpus, originally in English. It has been human-translated into eight other languages: Spanish, German, French, Portuguese, Hindi, Chinese, Japanese, and Turkish. The paper highlights that the Hindi and Turkish portions of the data are particularly under-resourced, with smaller subsets of intents and slots and very few labeled examples.

Key Insights and Ideas

  • Synergy is Key: Combining the cross-lingual representation power of mBERT with the few-shot learning capabilities of PNNs significantly boosts performance for low-resource languages.
  • Beyond High-Resource: The approach demonstrates that meaningful latent spaces learned in one language can indeed be generalized to unseen and under-resourced languages, challenging the notion that NLU is inherently limited to data-rich environments.
  • Few-Shot Advantage: Even with very limited data (e.g., 5-way, 10-shot), the mBERT+PNN approach outperforms baseline mBERT models, especially in the multilingual configuration.
  • Language Similarities Matter: The performance gains are more pronounced when transferring knowledge between linguistically similar languages (e.g., European languages) compared to dissimilar ones (e.g., Hindi/Turkish vs. European languages), suggesting that exploiting linguistic interrelationships is crucial.
  • Precision vs. Recall Trade-off: In some cases, the PNN approach leads to a reduction in recall but an improvement in precision, potentially by mitigating overlapping and annotation mismatch issues in the data.

Key Results

The paper reports significant improvements in both intent classification and slot-filling tasks, particularly for under-resourced languages.

  • Intent Classification: In the multilingual configuration, the mBERT+PNN (5w10s) model achieved up to a 9-point accuracy gain for under-resourced languages compared to the "target only" configuration. It also showed an average gain of one point compared to the best model in the multilingual zero-shot configuration.
  • Slot Filling: The mBERT+PNN (5w10s) model outperformed the baseline mBERT model by more than 4 F1 points in the Turkish task in the "target only" configuration. In the multilingual configuration, the approach consistently outperformed the baseline across all languages, with F1-score gains ranging from 2 to 5 points for slot filling.

Limitations and Future Work

  • Unbalanced Data: The MultiATIS++ corpus is highly unbalanced, with a wide variation in the number of training examples per class. This imbalance impacts model performance and is acknowledged as a factor influencing the observed results.
  • Multilingual Zero-Shot Performance: While the mBERT+PNN approach generally performs well, it sometimes falters in the multilingual zero-shot configuration for certain languages, though it still shows improvements for Hindi and Turkish.
  • Future Work: The authors suggest a promising direction for future research: adapting the few-shot setting to a joint approach for intent detection and slot filling. They note that performing these two tasks jointly has been shown to improve performance in other contexts.

Summary updated

BibTeX
@inproceedings{cattan2021cross,
  author = {Oralie Cattan and Sophie Rosset and Christophe Servan},
  title = {{On the cross-lingual transferability of multilingual prototypical models across NLU tasks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure