Abstract
Zhaojiang Lin, Zihan Liu, Genta Indra Winata, Samuel Cahyawijaya, Andrea Madotto, Yejin Bang, Etsuko Ishii, Pascale Fung. Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI.
2021.
Introduction
Personalized AI chatbots—agents capable of maintaining a consistent persona (e.g., "I am a jazz-loving vegan")—have primarily been developed for English. Creating high-quality conversational datasets in multiple languages is resource-intensive and expensive. Current approaches often rely on translation pipelines, which suffer from latency and error amplification, or cross-lingual transfer, which often fails to capture linguistic nuances. This paper introduces XPersona, a large-scale multilingual dataset, and evaluates various modeling strategies to bridge the gap in personalized conversational AI.
The XPersona Dataset
The core contribution of this work is the XPersona dataset, an extension of the English Persona-Chat. It covers seven languages: English (En), Chinese (Zh), French (Fr), Indonesian (Id), Italian (It), Korean (Ko), and Japanese (Jp).
While training sets were generated via machine translation and human refinement, the validation and test sets were fully human-annotated to ensure rigorous evaluation. Each dialogue is grounded in a specific persona consisting of 4–5 descriptive sentences. The refinement process involved humans identifying and correcting frequent translation errors (e.g., literal translations of idioms) to improve data quality.
Methods
The researchers evaluated two primary architectures: Encoder-Decoder models and Causal Decoders. They found that Causal Decoders are superior at maintaining context by treating the persona, dialogue history, and current query as a single coherent sequence.
To leverage multilingual data, models were initialized with M-BERT weights and trained jointly on all seven languages. This "multilingual training" approach allows a single model to serve multiple languages efficiently without needing separate language identifiers. The study also explored XNLG for zero-shot cross-lingual transfer, attempting to train on English and evaluate on other languages without target-specific training data.
Key Results and Insights
The study yielded several critical findings regarding model performance across languages:
- Multilingual Efficiency: Multilingual models (trained on all languages simultaneously) performed on par with, or better than, individual monolingual models. This suggests that a single unified model is a viable and efficient solution for global deployment.
- Architecture Matters: Causal Decoders (e.g., M-CausalBert) significantly outperformed Encoder-Decoder variants. For instance, in Italian, M-CausalBert achieved a perplexity of 15.12 compared to Bert2Bert's 50.36, highlighting the architectural advantage of causal decoding in dialogue.
- Human Preference: Using ACUTE-EVAL, human testers preferred the multilingual M-CausalBert over translation pipelines in nearly every language, citing higher "engagingness" and "humanness."
- The Cross-Lingual Gap: Zero-shot performance remains a significant challenge. Models trained only on English and tested on other languages (XNLG) performed significantly worse than those trained on target-language data, even if that data was machine-translated.
Limitations and Future Work
Despite its successes, the models still exhibit occasional repetition in long dialogues. Furthermore, while human-refined, the training data's machine-translated origins may still harbor subtle unnatural phrasings compared to organic conversation. Future research should focus on improving zero-shot transfer to support low-resource languages where high-quality translation is unavailable.
Summary updated