Introduction
Spoken Language Understanding (SLU) serves as the core intelligence for virtual assistants, enabling them to parse user requests into actionable data. This process involves two primary tasks: Intent Detection, which identifies the user's goal (e.g., "set alarm"), and Slot Filling, which extracts specific parameters (e.g., "five am"). Scaling these systems to support dozens of languages is a significant challenge in modern NLP. While data augmentation—such as translating English training data into target languages—is a common strategy, standard approaches often fail to leverage the semantic equivalence between an original utterance and its translated counterpart.
Methodology: Consistency Regularization
The HIT-SCIR team's winning approach at the MMNLU-22 competition centers on the principle of consistency regularization. The core idea is that a robust model should produce identical predictions for an utterance and its "twins"—versions of the same sentence modified through various augmentation techniques.
The framework utilizes a dual-pronged Hybrid Data Augmentation strategy:
- Subword Sampling: Using the SentencePiece algorithm to generate varied tokenization patterns for the same input string, forcing the model to be invariant to subword boundaries.
- Machine Translation (MT): Translating utterances into multiple languages to encourage cross-lingual semantic invariance.
To enforce this, the authors introduce a Consistency Loss based on Symmetrical Kullback-Leibler (KL) Divergence. This loss penalizes the model if the probability distributions for the original and augmented utterances diverge. While subword sampling is applied to both intent and slot tasks, MT is primarily used for intent detection due to the technical difficulties of maintaining precise slot alignment across translations.
Experimental Setup
The models were evaluated using the MASSIVE dataset, a comprehensive multilingual SLU benchmark covering 51 languages, 60 intent types, and 55 slot types. The team tested their approach under two paradigms:
- Full-dataset: Training on all available data across all languages.
- Zero-shot: Training exclusively on English data and testing on the remaining 50 languages.
Results and Impact
The proposed method achieved state-of-the-art results, securing 1st place in the MMNLU-22 competition's full-dataset track with an Exact Match Accuracy (EMA) of 49.65.
The impact was even more pronounced in the zero-shot setting. By enforcing semantic consistency, the model was better able to bridge the gap between English and unseen languages. As shown in the results below, the consistency regularization framework provided a massive boost over standard baselines.
In the zero-shot track, the method improved the XLM-Align baseline by 21.27 points in EMA. Ablation studies further confirmed that both subword sampling and the consistency loss were critical drivers of these performance gains.
Discussion and Future Work
A key takeaway from this research is that data augmentation should be viewed as a constraint for enforcing invariance rather than just additional supervision. However, the "Slot Alignment Bottleneck" remains a challenge; ensuring that slots like "time" or "date" map perfectly across languages with different word orders requires complex engineering. Future work may explore more sophisticated alignment techniques or knowledge distillation from larger, more computationally expensive models to further refine these efficient base architectures.
Summary updated