Abstract
We present a systematic study on multilingual and cross-lingual intent detection (ID) from spoken data. The study leverages a new resource put forth in this work, termed MInDS-14, a first training and evaluation resource for the ID task with spoken data. It covers 14 intents extracted from a commercial system in the e-banking domain, associated with spoken examples in 14 diverse language varieties. Our key results indicate that combining machine translation models with state-of-the-art multilingual sentence encoders (e.g., LaBSE) yield strong intent detectors in the majority of target languages covered in MInDS-14, and offer comparative analyses across different axes: e.g., translation direction, impact of speech recognition, data augmentation from a related domain. We see this work as an important step towards more inclusive development and evaluation of multilingual ID from spoken data, hopefully in a much wider spectrum of languages compared to prior work.
Introduction
Intent Detection (ID) is a core component of task-oriented dialogue systems, such as virtual banking assistants, responsible for identifying user goals (e.g., "freeze a card" or "check a balance"). Despite its importance, current ID research faces significant challenges: an overwhelming focus on English-only benchmarks, a reliance on text-based datasets that ignore the noise of real-world voice interactions (ASR errors, background noise), and a lack of multilingual data in specialized domains like e-banking.
To address these gaps, we introduce MInDS-14, the first diverse multilingual dataset for intent detection specifically designed for spoken data. Our study reveals that even straightforward "ASR-then-translate" pipelines, when paired with robust multilingual encoders like LaBSE, can achieve over 95% accuracy across 14 different language varieties.
Methodology
We employ a feature-based approach to intent detection. Rather than computationally intensive fine-tuning, we utilize fixed, off-the-shelf multilingual sentence encoders to generate utterance embeddings, which are then classified by a simple Multi-Layer Perceptron (MLP).
We evaluated three primary training strategies:
- Translate-to-EN: Non-English speech transcripts are translated into English and processed by an English-trained model.
- Target-only: Models are trained exclusively on data from the specific target language.
- Multilingual: A single model is trained on a combined dataset of all available languages.
Additionally, we explored data augmentation by using $n$-best ASR hypotheses and multiple translation services (Google Translate and DeepL) to increase training diversity and robustness to ASR noise.
The MInDS-14 Dataset
MInDS-14 covers 14 language varieties, including three English dialects (AU, GB, US), European languages (FR, IT, ES, PT, DE, NL), Slavic languages (RU, PL, CS), and Asian languages (KO, ZH). The dataset consists of 14 intents derived from a real-world commercial e-banking system, including both original audio recordings and their corresponding ASR transcripts. To facilitate cross-lingual transfer testing, we also utilize a small auxiliary set of 660 English utterances (AUX-EN).
Results and Analysis
Our experiments reveal several key insights into multilingual ID performance. The Language-Agnostic BERT Sentence Embedding (LaBSE) consistently outperformed the Multilingual Universal Sentence Encoder (mUSE), demonstrating superior robustness for cross-lingual tasks.
A significant finding is the strength of the "ASR-then-translate" pipeline. Using LaBSE in the "translate-to-EN" setup achieved an average accuracy of 95.9% across all 14 languages, often surpassing models trained directly on native target-language data. Performance remained consistent across most languages, with slight dips in typologically distant languages like Korean (94.0%) and Polish (93.7%).
Furthermore, we found that in-domain data is crucial; even small amounts of banking-specific data significantly outperformed larger general-purpose datasets. Data augmentation via $n$-best ASR hypotheses and dual translation services also provided marginal but consistent gains, enhancing the model's resilience to transcription errors.
Conclusion
While MInDS-14 represents a major step forward, it currently focuses on high-resource languages with established ASR and machine translation support. Future work will aim to expand this benchmark to low-resource languages, investigate more complex dialogue domains, and evaluate performance under more challenging acoustic conditions.
Summary updated