HIT-SCIR at MMNLU-22: Consistency Regularization for Multilingual Spoken Language Understanding

Bo Zheng, Zhouyang Li, Fuxuan Wei, Qiguang Chen, Libo Qin, Wanxiang ChearXiv 2023 · DOI · arXiv

Introduction

Spoken Language Understanding (SLU) serves as the core intelligence for virtual assistants, enabling them to parse user requests into actionable data. This process involves two primary tasks: Intent Detection, which identifies the user's goal (e.g., "set alarm"), and Slot Filling, which extracts specific parameters (e.g., "five am"). Scaling these systems to support dozens of languages is a significant challenge in modern NLP. While data augmentation—such as translating English training data into target languages—is a common strategy, standard approaches often fail to leverage the semantic equivalence between an original utterance and its translated counterpart.

Methodology: Consistency Regularization

The HIT-SCIR team's winning approach at the MMNLU-22 competition centers on the principle of consistency regularization. The core idea is that a robust model should produce identical predictions for an utterance and its "twins"—versions of the same sentence modified through various augmentation techniques.

Overview of the proposed framework featuring hybrid data augmentation and consistency regularization.

The framework utilizes a dual-pronged Hybrid Data Augmentation strategy:

  1. Subword Sampling: Using the SentencePiece algorithm to generate varied tokenization patterns for the same input string, forcing the model to be invariant to subword boundaries.
  2. Machine Translation (MT): Translating utterances into multiple languages to encourage cross-lingual semantic invariance.

To enforce this, the authors introduce a Consistency Loss based on Symmetrical Kullback-Leibler (KL) Divergence. This loss penalizes the model if the probability distributions for the original and augmented utterances diverge. While subword sampling is applied to both intent and slot tasks, MT is primarily used for intent detection due to the technical difficulties of maintaining precise slot alignment across translations.

Experimental Setup

The models were evaluated using the MASSIVE dataset, a comprehensive multilingual SLU benchmark covering 51 languages, 60 intent types, and 55 slot types. The team tested their approach under two paradigms:

  • Full-dataset: Training on all available data across all languages.
  • Zero-shot: Training exclusively on English data and testing on the remaining 50 languages.

Results and Impact

The proposed method achieved state-of-the-art results, securing 1st place in the MMNLU-22 competition's full-dataset track with an Exact Match Accuracy (EMA) of 49.65.

The impact was even more pronounced in the zero-shot setting. By enforcing semantic consistency, the model was better able to bridge the gap between English and unseen languages. As shown in the results below, the consistency regularization framework provided a massive boost over standard baselines.

Performance comparison in the zero-shot cross-lingual setting on the MASSIVE dataset.

In the zero-shot track, the method improved the XLM-Align baseline by 21.27 points in EMA. Ablation studies further confirmed that both subword sampling and the consistency loss were critical drivers of these performance gains.

Discussion and Future Work

A key takeaway from this research is that data augmentation should be viewed as a constraint for enforcing invariance rather than just additional supervision. However, the "Slot Alignment Bottleneck" remains a challenge; ensuring that slots like "time" or "date" map perfectly across languages with different word orders requires complex engineering. Future work may explore more sophisticated alignment techniques or knowledge distillation from larger, more computationally expensive models to further refine these efficient base architectures.

Summary updated

Abstract

Multilingual spoken language understanding (SLU) consists of two sub-tasks, namely intent detection and slot filling. To improve the performance of these two sub-tasks, we propose to use consistency regularization based on a hybrid data augmentation strategy. The consistency regularization enforces the predicted distributions for an example and its semantically equivalent augmentation to be consistent. We conduct experiments on the MASSIVE dataset under both full-dataset and zero-shot settings. Experimental results demonstrate that our proposed method improves the performance on both intent detection and slot filling tasks. Our system\footnote{The code will be available at \url{https://github.com/bozheng-hit/MMNLU-22-HIT-SCIR}.} ranked 1st in the MMNLU-22 competition under the full-dataset setting.
BibTeX
@misc{zheng2023hit,
  author = {Bo Zheng and Zhouyang Li and Fuxuan Wei and Qiguang Chen and Libo Qin and Wanxiang Che},
  title = {{HIT-SCIR at MMNLU-22: Consistency Regularization for Multilingual Spoken Language Understanding}},
  year = {2023},
  eprint = {2301.02010},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.02010},
}
Figure