TL;DR
This paper introduces HC2L, a novel hybrid and cooperative contrastive learning framework for zero-shot cross-lingual spoken language understanding (SLU). By integrating supervised contrastive learning with existing unsupervised methods, HC2L significantly improves intent detection and slot filling accuracy across multiple languages, achieving new state-of-the-art results.
What Problem Does This Paper Address?
Spoken Language Understanding (SLU) is crucial for task-oriented dialogue systems, involving two key tasks: intent detection (classifying the user's goal) and slot filling (extracting specific information like dates or locations). While significant progress has been made, current methods often require vast amounts of labeled data, which is impractical for many languages. Zero-shot cross-lingual SLU aims to bridge this gap by transferring knowledge from high-resource languages (like English) to low-resource ones without requiring target-language training data.
The current state-of-the-art, exemplified by GL-CLEF, uses cross-lingual unsupervised contrastive learning (CL) to align semantic representations between an utterance and its code-switched version. However, this approach overlooks valuable label information (intent and slot labels). This oversight leads to two issues: (1) utterances with similar labels might be incorrectly separated, and (2) utterances with different labels might not be sufficiently pushed apart in the semantic space. This paper addresses this limitation by proposing a method that leverages label information more effectively.
Methods
The core contribution of this paper is the Hybrid and Cooperative Contrastive Learning (HC2L) framework. HC2L builds upon the existing cross-lingual unsupervised contrastive learning (UCL) by introducing three novel supervised contrastive learning mechanisms:
- Source Language Supervised CL: This mechanism uses intent and slot labels from the source language to learn label-aware semantic structures, enhancing the source language representations. It includes both single-task (intent or slot) and joint-task (intent and slot) scenarios.
- Cross-lingual Supervised CL: This method transfers label-aware semantic knowledge from the source language to target languages. It aligns source language representations with multilingual views (code-switched versions) based on shared labels.
- Multilingual Supervised CL: This approach leverages the multilingual views of utterances, which already contain target language semantics. It performs pseudo-target language supervised CL by aligning these views with their corresponding labels, effectively learning better target language semantics.
A key aspect of HC2L is the cooperative nature of these four CL mechanisms. The input semantics for one mechanism are enhanced by others, creating a virtuous cycle that leads to more consistent and discriminative representations. The framework also incorporates sample queues to store previously encoded features and labels, facilitating the contrastive learning process.
Datasets
The experiments were conducted on the MultiATIS++ dataset, a multilingual benchmark comprising 9 languages: English (en), Spanish (es), Portuguese (pt), German (de), French (fr), Chinese (zh), Japanese (ja), Hindi (hi), and Turkish (tr). This dataset includes 18 intent classes and 84 slot labels. The training set is in English, with validation and testing sets provided for each language.
Key Insights and Ideas
- Labels are Crucial for Cross-lingual SLU: The paper strongly argues that ignoring intent and slot labels in contrastive learning is a significant drawback. Incorporating label information leads to more accurate and discriminative semantic representations.
- Cooperation is Key: The "hybrid and cooperative" aspect is central. The four contrastive learning mechanisms (one unsupervised, three supervised) don't just work in parallel; they actively enhance each other, leading to synergistic improvements.
- Pseudo-Target Supervision: The multilingual supervised CL mechanism ingeniously uses multilingual views (which have ground-truth labels from the source language) to perform supervised learning on target languages, overcoming the lack of direct target-language training data.
- Improved Semantic Separation: Visualizations (t-SNE) demonstrate that HC2L creates more distinct and well-separated clusters for different intent classes, both within the source language and across target languages, compared to previous methods.
Key Results
- State-of-the-Art Performance: HC2L achieved significant improvements across all 9 languages, outperforming the previous state-of-the-art model, GL-CLEF, by approximately 10.1% in overall accuracy.
- Substantial Gains in Low-Resource Languages: The model showed particularly sharp improvements in languages with inferior performance, such as Hindi (hi), where it achieved nearly a 50% relative improvement in overall accuracy.
- Effectiveness of Supervised CL: Ablation studies showed that the proposed supervised contrastive learning mechanisms contributed more than the unsupervised CL alone.
- Computational Efficiency: HC2L required slightly less GPU memory (8.7% reduction) than GL-CLEF, with a marginal increase in training time (6.1%).
Limitations and Future Work
- Hyperparameter Sensitivity: The paper notes that the performance is sensitive to hyperparameter choices that balance the different contrastive learning objectives (β₁, βs, βJ, γ₁, γ₂). The optimal values were found through extensive tuning.
- Foundation Model Dependence: While HC2L shows strong performance, its effectiveness is still dependent on the underlying pre-trained multilingual encoder (e.g., mBERT, XLM-R). The authors suggest that their method can be applied to further improve performance with future, stronger encoders.
- Future Work: The authors suggest that their model-agnostic approach can be applied to improve performance with other pre-trained multilingual encoders, implying that further advancements in encoder architectures could lead to even better results with HC2L.
Summary updated