End-to-End Slot Alignment and Recognition for Cross-Lingual NLU

Weijia Xu, Batool Haider, Saab Mansour

Abstract

Natural language understanding (NLU) in the context of goal-oriented dialog systems typically includes intent classification and slot labeling tasks. Existing methods to expand an NLU system to new languages use machine translation with slot label projection from source to the translated utterances, and thus are sensitive to projection errors. In this work, we propose a novel end-to-end model that learns to align and predict target slot labels jointly for cross-lingual transfer. We introduce MultiATIS++, a new multilingual NLU corpus that extends the Multilingual ATIS corpus to nine languages across four language families, and evaluate our method using the corpus. Results show that our method outperforms a simple label projection method using fast-align on most languages, and achieves competitive performance to the more complex, state-of-the-art projection method with only half of the training time. We release our MultiATIS++ corpus to the community to continue future research on cross-lingual NLU.

Introduction

Building Natural Language Understanding (NLU) systems for new languages is a resource-intensive task. Traditional approaches often rely on translating high-quality English data into a target language and using "label projection" to map English slot labels (e.g., dates, locations) onto translated words. However, this process is brittle, frequently relying on external "hard-alignment" tools like fast-align that struggle with structural differences between languages. Misalignments lead to noisy training data and degraded model performance.

To address this, we propose a soft-alignment model that learns these mappings end-to-end. To evaluate this approach, we introduce MultiATIS++, an expansion of the classic Airline Travel Information System (ATIS) dataset. MultiATIS++ covers nine languages across four distinct language families (Indo-European, Sino-Tibetan, Japonic, and Altaic), providing a rigorous benchmark for cross-lingual transfer.

Methodology

The proposed model is built upon a multilingual BERT (mBERT) encoder and integrates several key components to facilitate joint learning of intent and slots.

Architecture of the proposed end-to-end soft-alignment model.

Joint Learning and Soft-Alignment

Instead of a multi-step pipeline, the model simultaneously performs intent classification and slot filling. Central to the architecture is an attention-based alignment module. Rather than forcing a restrictive one-to-one word mapping, the model uses attention to "look" at the translated utterance and aggregate information from multiple words to represent a single slot from the source.

Reconstruction Objective

To ensure the attention mechanism captures meaningful semantic connections, we incorporate a reconstruction loss. The model is tasked with rebuilding the original English utterance using only the attention outputs from the target language. This acts as a powerful regularizer, forcing the model to preserve semantic meaning during alignment.

Experimental Results

Evaluation on the MultiATIS++ corpus demonstrates the effectiveness of the MT+soft-align approach across diverse linguistic structures.

Performance comparison of intent accuracy and slot F1 across French and Chinese datasets.

Key Findings

  • Superior Performance: The soft-alignment method outperformed the standard MT+fast-align baseline on most languages. On Turkish, a morphologically rich and distant language, it improved the Slot F1 score by 16.9%.
  • Efficiency: The model achieves performance competitive with state-of-the-art methods (like Translate-Match-Project) while requiring only half the training time.
  • Multilingual Synergy: Training on all nine languages simultaneously using mBERT significantly boosted performance for low-resource languages like Hindi and Turkish compared to training on them individually.

Conclusion

This work demonstrates that end-to-end soft alignment is a robust and efficient alternative to traditional label projection for cross-lingual NLU. By releasing MultiATIS++, we provide the community with a diverse dataset to further explore these challenges. Future work includes exploring more powerful encoders like XLM-R and reducing the reliance on initial machine translation.

Summary updated

Abstract

Natural language understanding (NLU) in the context of goal-oriented dialog systems typically includes intent classification and slot labeling tasks. Existing methods to expand an NLU system to new languages use machine translation with slot label projection from source to the translated utterances, and thus are sensitive to projection errors. In this work, we propose a novel end-to-end model that learns to align and predict target slot labels jointly for cross-lingual transfer. We introduce MultiATIS++, a new multilingual NLU corpus that extends the Multilingual ATIS corpus to nine languages across four language families, and evaluate our method using the corpus. Results show that our method outperforms a simple label projection method using fast-align on most languages, and achieves competitive performance to the more complex, state-of-the-art projection method with only half of the training time. We release our MultiATIS++ corpus to the community to continue future research on cross-lingual NLU.
BibTeX
@inproceedings{xu2020end,
  author = {Weijia Xu and Batool Haider and Saab Mansour},
  title = {{End-to-End Slot Alignment and Recognition for Cross-Lingual NLU}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=V498O1orLYQ},
}
Figure