TL;DR
This paper introduces HC2L, a novel hybrid contrastive learning framework for zero-shot cross-lingual spoken language understanding (SLU). By combining unsupervised and multiple supervised contrastive learning strategies, HC2L effectively leverages label information to improve semantic alignment across languages, achieving state-of-the-art results on a multilingual benchmark.
What Problem Does This Paper Address?
Spoken Language Understanding (SLU) is crucial for task-oriented dialogue systems, involving intent detection and slot filling. While zero-shot cross-lingual SLU aims to transfer knowledge from high-resource to low-resource languages without extensive data annotation, existing state-of-the-art methods, like GL-CLEF, primarily rely on cross-lingual unsupervised contrastive learning. This approach, while effective for semantic alignment, overlooks valuable intent and slot label information. The paper argues that this oversight leads to suboptimal semantic alignment, as it fails to leverage label-aware structures that could enhance both source and target language understanding. The core problem is how to effectively integrate label information into contrastive learning for robust cross-lingual SLU.
Methods
The proposed HC2L (Hybrid and Cooperative Contrastive Learning) framework builds upon the existing cross-lingual unsupervised contrastive learning (CL) by introducing three novel supervised CL mechanisms:
- Source Language Supervised CL: This mechanism leverages intent and slot labels from the source language to learn label-aware semantic structures. It includes both single-task (intent/slot) and joint-task (intent+slot) scenarios.
- Cross-lingual Supervised CL: This component transfers knowledge from the source to target languages by aligning source language semantics with multilingual views (code-switched data) in a label-aware manner.
- Multilingual Supervised CL: This mechanism aligns multilingual views that share similar labels, effectively performing pseudo-supervised learning on target languages using their existing labels.
A key innovation is the "cooperative" aspect: the input semantics for one CL mechanism are enhanced by others, creating a virtuous cycle during training. This allows the four CL mechanisms (one unsupervised, three supervised) to work together to learn more consistent and discriminative representations. The framework also utilizes sample queues to store previously encoded features and labels, facilitating the contrastive learning process.
Datasets
The primary dataset used for training and evaluation is MultiATIS++. This benchmark comprises 9 languages (English, Spanish, Portuguese, German, French, Chinese, Japanese, Hindi, and Turkish) and includes 18 intent classes and 84 slot labels. The training set is in English, with validation and testing sets provided for each language.
Key Insights and Ideas
- Label Information is Crucial: The paper strongly advocates for the importance of intent and slot labels in contrastive learning for SLU, arguing that their omission by previous methods limits performance.
- Cooperative Learning: The core idea of "cooperative" contrastive learning, where different CL mechanisms mutually enhance each other, is central to HC2L's success. This synergistic approach leads to better representations than individual components.
- Pseudo-Supervised Learning for Target Languages: Multilingual supervised CL offers a clever way to leverage label information for target languages, even when direct supervision is unavailable, by using multilingual views.
- Joint-Task Learning: The inclusion of joint-task (intent+slot) supervised CL, even with self-constructed labels, is shown to be effective in modeling the interdependencies between these two crucial SLU tasks.
Key Results
HC2L achieves state-of-the-art performance on the MultiATIS++ benchmark across 9 languages.
- Overall Accuracy: The model obtains approximately 10% average improvement in overall accuracy compared to the previous best-performing model, GL-CLEF.
- Language-Specific Improvements: HC2L shows particularly sharp improvements on languages with inferior performance, such as nearly 50% relative improvement for Hindi (hi) in overall accuracy.
- Ablation Studies:
- Supervised CL mechanisms contribute more than unsupervised CL alone.
- All three proposed supervised CL mechanisms (Source, Cross-lingual, Multilingual) bring significant improvements.
- The cooperative effect of all four CL mechanisms (UCL + SCL) yields the best performance.
- Visualization: TSNE visualizations demonstrate that HC2L generates more discriminative intent clusters and more consistent multilingual representations compared to GL-CLEF.
Limitations and Future Work
- Computational Cost: While HC2L uses slightly less GPU memory than GL-CLEF, it incurs a slightly higher training time per epoch (35s vs. 33s).
- Model-Agnosticism: The authors suggest that their method is model-agnostic and can be applied to further improve performance with stronger pre-trained multilingual encoders, indicating a clear path for future research.
- Hyperparameter Sensitivity: The paper analyzes the impact of hyperparameter tuning for the supervised CL components, showing that performance peaks at specific values, implying careful tuning is necessary.
Summary updated