Data Augmentation and Learned Layer Aggregation for Improved Multilingual Language Understanding in Dialogue

Evgeniia Razumovskaia, Ivan Vulic, Anna Korhonen

Introduction

Scaling task-oriented dialogue systems—the technology powering virtual assistants—to support dozens of languages remains a significant challenge in AI. The primary bottleneck is not the underlying code, but the scarcity of annotated data. Manually labeling "intents" (user goals) and "slots" (specific entities like dates or locations) for every new language is both prohibitively expensive and time-consuming.

Researchers from the University of Cambridge address this by introducing two novel methods: Multi-SentAugment and LayerAgg. These techniques optimize performance by leveraging clever data mining and improving how models navigate the internal representations of Transformer architectures.

The Challenge of Multilingual NLU

Natural Language Understanding (NLU) in dialogue systems typically involves two concurrent tasks: Intent Detection (e.g., identifying that a user wants to "book a flight") and Slot Filling (e.g., extracting "London" as the origin and "NYC" as the destination). While models perform exceptionally well in English, their accuracy drops significantly when applied to languages with limited labeled data, such as Turkish or Hindi.

Examples of intent and slot annotation in English and Russian.

Current state-of-the-art approaches often rely on "zero-shot" transfer, where a model trained on English is tested on a target language. However, these models frequently fail to generalize because they ignore useful low-level linguistic information stored in the middle layers of the network.

Methodology

The proposed strategy improves performance at both the data and architectural levels:

1. Multi-SentAugment (Data Augmentation)

This is a weakly supervised pipeline designed to expand small labeled datasets. Using a multilingual sentence encoder (such as LASER), the system identifies similar, unannotated sentences within massive web-crawled corpora (CC-100). A "teacher" model then automatically labels these new sentences, creating a larger synthetic dataset to train a "student" model.

2. LayerAgg (Architecture Optimization)

Standard practices typically utilize only the final output layer of a Transformer (like mBERT) for classification. However, research indicates that lower layers often contain superior lexical and syntactic information. LayerAgg introduces a small, trainable Transformer "aggregator" that learns to weight and combine information from all layers. This allows the model to retain linguistic nuances essential for cross-lingual slot filling.

Key Results and Insights

The researchers evaluated these methods on major benchmarks, including MultiATIS++ and xSID, across 16 diverse languages.

Performance comparison across languages for intent classification and slot labeling in few-shot and full-data scenarios.
  • Synergistic Gains: Multi-SentAugment and LayerAgg are complementary. While augmentation increases input variety, LayerAgg improves internal representations. The combination yields the highest performance, particularly in few-shot scenarios (e.g., using only 10 examples per intent).
  • Zero-Shot Performance: On MultiATIS++, LayerAgg improved mBERT's intent accuracy from 83.4% to 84.3% and slot filling F1 from 57.3% to 60.1%. Using the more powerful XLM-R model, LayerAgg reached 86.3% intent accuracy and 64.6% slot F1.
  • Layer Importance: LayerAgg demonstrates that the top layer is not always optimal for cross-lingual transfer. By accessing lower layers, the model captures "decontextualized" lexical information that remains more stable across different languages.

Conclusion

The combination of smarter data augmentation and learned layer aggregation significantly boosts the efficiency of multilingual dialogue systems. While challenges remain—such as tokenization discrepancies in complex scripts like Chinese—these methods provide a robust framework for expanding AI capabilities to low-resource languages.

Code available at: github.com/cambridgeltl/MultiSentAugment_LayerAgg

Summary updated

Abstract

Scaling dialogue systems to a multitude of domains, tasks and languages relies on costly and time-consuming data annotation for different domain-task-language configurations. The annotation efforts might be substantially reduced by the methods that generalise well in zero- and few-shot scenarios, and also effectively leverage external unannotated data sources (e.g., Web-scale corpora). We propose two methods to this aim, offering improved dialogue natural language understanding (NLU) across multiple languages: 1) Multi-SentAugment, and 2) LayerAgg. Multi-SentAugment is a self-training method which augments available (typically few-shot) training data with similar (automatically labelled) in-domain sentences from large monolingual Web-scale corpora. LayerAgg learns to select and combine useful semantic information scattered across different layers of a Transformer model (e.g., mBERT); it is especially suited for zero-shot scenarios as semantically richer representations should strengthen the model’s cross-lingual capabilities. Applying the two methods with state-of-the-art NLU models obtains consistent improvements across two standard multilingual NLU datasets covering 16 diverse languages. The gains are observed in zero-shot, few-shot, and even in full-data scenarios. The results also suggest that the two methods achieve a synergistic effect: the best overall performance in few-shot setups is attained when the methods are used together.
BibTeX
@inproceedings{razumovskaia2022data,
  author = {Evgeniia Razumovskaia and Ivan Vulic and Anna Korhonen},
  title = {{Data Augmentation and Learned Layer Aggregation for Improved Multilingual Language Understanding in Dialogue}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.160},
}
Figure