End-to-End Slot Alignment and Recognition for Cross-Lingual NLU

Weijia Xu, Batool Haider, Saab Mansour

Abstract

Researchers from Amazon AI and the University of Maryland have introduced a novel end-to-end model for cross-lingual Natural Language Understanding (NLU) that learns to align and recognize "slots" (e.g., city names, dates) without relying on external alignment tools. To support this work, they released MultiATIS++, an expansion of the classic ATIS dataset covering nine languages across four language families. Their proposed soft-alignment method achieves state-of-the-art results while being significantly more efficient than traditional pipelines.

Introduction

Building goal-oriented dialog systems like Alexa or Siri for new languages is a resource-intensive task. NLU typically involves two core components: Intent Classification (identifying the user's goal) and Slot Filling (extracting specific entities).

A common strategy for scaling to new languages is to translate high-quality English data and "project" labels onto the translated text. However, this projection usually depends on external word-alignment tools (e.g., fast-align), which often struggle with linguistically distant languages and introduce errors that degrade model performance. Furthermore, these methods often rely on complex, hand-engineered features that do not generalize well to low-resource settings.

Proposed Architecture: End-to-End Soft-Alignment

The authors propose a model that treats label projection as an internal latent variable within the neural network.

The end-to-end architecture featuring soft-alignment via attention and a source reconstruction objective.

Key Components:

  1. Shared Encoder: The model uses a shared encoder (LSTM or Multilingual BERT) to represent both source (English) and target (translated) utterances.
  2. Soft-Alignment via Attention: Instead of a rigid one-to-one mapping, the model employs an attention mechanism to learn a "soft" mapping between the translated utterance and the labeled slots in the original English sentence.
  3. Augmented Reconstruction Objective: To ensure the attention mechanism learns semantically meaningful alignments, the model is trained to reconstruct the original English utterance from the attention-weighted representations of the target text. This regularizes the latent alignment.
  4. Efficient Inference: During deployment, the attention and reconstruction modules are discarded. The model directly predicts intents and slots from the target language, ensuring high speed.

The MultiATIS++ Dataset

A major contribution of this work is the release of MultiATIS++, which significantly expands the existing Multilingual ATIS corpus.

  • Expanded Scope: It adds Spanish, German, French, Portuguese, Chinese, and Japanese to the original English, Hindi, and Turkish set.
  • Scale: The dataset includes 37,084 training and 7,859 test examples across nine languages and four language families (Indo-European, Sino-Tibetan, Japonic, and Altaic).
  • High Quality: Utterances were translated by professional native speakers instructed to maintain realistic spoken modalities.

Results and Key Insights

The proposed MT+soft-align model demonstrates superior performance across various settings, particularly in zero-shot and few-shot scenarios.

Intent accuracy and Slot F1 scores on French and Chinese across different target data sizes.

Performance Highlights:

  • Robustness to Language Distance: The soft-alignment method excels where traditional tools fail. For example, on Turkish—a morphologically rich language—the model outperformed the complex Translate-Match-Project (TMP) method by 16.9% in Slot F1.
  • Efficiency: The end-to-end approach achieves performance on par with state-of-the-art systems like TMP while being twice as fast to train.
  • Few-Shot Learning: As shown in the performance plots, the model effectively leverages small amounts of target-language data, outperforming traditional projection methods that are often limited by initial alignment errors.

Conclusion

By integrating alignment directly into the training objective, this research provides a more robust and efficient path for scaling NLU systems to diverse languages. While the model currently relies on machine translation for initial training pairs, future work may explore reducing this dependency further and utilizing more advanced multilingual encoders like XLM-R.

Summary updated

BibTeX
@inproceedings{xu2020end,
  author = {Weijia Xu and Batool Haider and Saab Mansour},
  title = {{End-to-End Slot Alignment and Recognition for Cross-Lingual NLU}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=V498O1orLYQ},
}
Figure