Addressee and Response Selection for Multilingual Conversation

Motoki Sato, Hiroki Ouchi, Yuta TsuboiCOLING 2018

TL;DR

This paper tackles the challenge of building conversational AI systems that can understand and respond in multiple languages, especially for low-resource languages. They introduce a new task called Multilingual Addressee and Response Selection (M-ARS) and propose novel knowledge transfer methods, including a W-GAN based approach, that effectively leverage high-resource language data to improve performance on low-resource languages.

What Problem Does This Paper Address?

Developing conversational systems that can converse fluently in many languages is a significant challenge. Existing models often require vast amounts of training data, which is readily available for high-resource languages like English but scarce for many others. This creates a "data divide," limiting the applicability of advanced conversational AI to a subset of the world's languages. The paper addresses this by focusing on how to transfer knowledge from high-resource languages to low-resource ones within the context of addressee and response selection, a crucial component of multi-turn conversations. They specifically aim to build systems that can either be language-specific (one model per language) or language-invariant (a single model for all languages).

Methods

The paper proposes several methods for multilingual addressee and response selection, categorized into language-specific and language-invariant approaches.

  • Multilingual Embedding Replacement: This basic method trains a model on high-resource language data and then replaces the language-specific word embeddings with those of the target low-resource language during inference. This requires no target language data.
  • Fine-Tuning: This method first pre-trains a model on high-resource language data and then fine-tunes it on the limited low-resource language data. This leverages the initial knowledge gained from the high-resource data.
  • Joint Loss Training: To build language-invariant models and avoid "catastrophic forgetting" (where a model forgets previously learned tasks when learning new ones), this method jointly trains the model on data from all languages simultaneously using a combined loss function.
  • Multi-Language Adversarial Training (W-GAN): This approach uses a Wasserstein Generative Adversarial Network (W-GAN) framework to learn language-invariant features. The goal is to match the distributions of feature representations between high-resource and low-resource languages, making the model more robust across different languages. This method is combined with the joint loss training.

Datasets

The authors introduce a new multilingual conversation corpus and dataset for M-ARS. This dataset was created by crawling and preprocessing Ubuntu IRC logs in 12 languages. They then created an M-ARS dataset by identifying ground-truth addressees and responses. The dataset is partitioned into training, development, and test sets. For experiments, they focus on English as the high-resource language and Italian, Croatian, German, and Portuguese as low-resource languages.

Key Insights and Ideas

  • Knowledge Transfer is Crucial: The core idea is that high-resource language data can significantly compensate for the lack of data in low-resource languages for conversational AI tasks.
  • W-GAN for Language Invariance: The W-GAN based adversarial training method proved particularly effective for learning language-invariant features, leading to the best performance in multi-language adaptation scenarios. This suggests that matching feature distributions is a powerful way to achieve cross-lingual generalization.
  • Fine-tuning Outperforms Simple Replacement: Fine-tuning a pre-trained model on target language data generally yields better results than simply replacing word embeddings, highlighting the benefit of adapting the model to the specific nuances of the target language.
  • Multilingual Embeddings Offer a Baseline: Even without target language data, the multilingual embedding replacement method (ENONLY) achieved performance significantly better than chance, indicating the inherent value of semantically aligned multilingual embeddings.
  • More Agents, Lower Accuracy: As the number of agents in a conversation context increases, the accuracy of addressee and response selection tends to decrease for all methods, but WGAN showed more stable performance in these complex scenarios.

Key Results

  • In single-language adaptation, the WGAN method achieved the best scores across most metrics, outperforming other methods like FINETUNE and TRGONLY.
  • In multi-language adaptation (two- and five-language), WGAN consistently outperformed JOINT and other methods, demonstrating its effectiveness in building a single unified model for multiple languages. For instance, in five-language adaptation, WGAN achieved the highest ADR-RES accuracy (23.39% for |R|=10).
  • The ENONLY method, which uses multilingual embeddings without any target language training data, achieved higher scores than chance-level, showing the utility of pre-trained multilingual embeddings.
  • Data augmentation using Neural Machine Translation (NMT) to translate high-resource language data to low-resource language data, when combined with knowledge transfer methods like FINETUNE, JOINT, and WGAN, further boosted performance.

Limitations and Future Work

  • Data Augmentation Noise: While NMT-based data augmentation is beneficial, the translations can be noisy, and preparing parallel corpora for NMT can be costly.
  • Evaluation Cost: The authors acknowledge that human judgment is costly and impractical for evaluating multilingual systems, especially for low-resource languages. They opted for accuracy-based metrics as a practical choice.
  • Future Work: The authors suggest extending their work to multilingual response generation, which is a more complex task than selection. They also highlight the interesting possibility of evaluating their language-invariant systems in code-mixing situations, where multiple languages are used within the same conversation.

Summary updated

BibTeX
@inproceedings{sato2018addressee,
  author = {Motoki Sato and Hiroki Ouchi and Yuta Tsuboi},
  title = {{Addressee and Response Selection for Multilingual Conversation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure