Abstract
We present a systematic study on multilingual and cross-lingual intent detection from spoken data. The study leverages a new resource put forth in this work, termed MInDS-14, a first training and evaluation resource for the intent detection task with spoken data. It covers 14 intents extracted from a commercial system in the e-banking domain, associated with spoken examples in 14 diverse language varieties. Our key results indicate that combining machine translation models with state-of-the-art multilingual sentence encoders (e.g., LaBSE) can yield strong intent detectors in the majority of target languages covered in MInDS-14, and offer comparative analyses across different axes: e.g., zero-shot versus few-shot learning, translation direction, and impact of speech recognition. We see this work as an important step towards more inclusive development and evaluation of multilingual intent detectors from spoken data, in a much wider spectrum of languages compared to prior work.
Introduction
Intent detection—the task of classifying a user's goal, such as "check balance" or "freeze card"—is the backbone of task-oriented dialogue systems. Despite its importance, current research has two major blind spots: an overwhelming focus on English and a reliance on clean, text-based datasets. Real-world applications, particularly in telephony, must process spoken data which introduces noise and Automatic Speech Recognition (ASR) errors. This paper addresses these gaps by introducing MInDS-14, a multilingual spoken dataset, and evaluating state-of-the-art cross-lingual models on this challenging data.
The MInDS-14 Dataset
MInDS-14 is the first resource specifically designed for multilingual intent detection from spoken data. It encompasses 14 distinct intents within the e-banking domain, spanning 14 language varieties including various dialects of English (AU, GB, US), Germanic, Romance, Slavic, and Asian languages. The data was collected via crowdsourcing using both phone-based assistants and online recording software to capture realistic telephony noise. Each language contains approximately 500–700 examples, providing a robust benchmark for both native and cross-lingual evaluation.
Methodology
The study employs a feature-based classification approach using fixed, pretrained multilingual sentence encoders: mUSE (Multilingual Universal Sentence Encoder) and LaBSE (Language-agnostic BERT Sentence Embedding). These encoders generate embeddings that are classified by a lightweight 2-layer Multi-Layer Perceptron (MLP).
The researchers explored several experimental pipelines to understand the impact of translation and data availability:
- Translate-to-EN: Transcribing speech and translating it to English before classification.
- Target-only: Training and testing directly on the native language.
- Multilingual: Training on a combined pool of all available languages.
Results and Key Insights
The experimental results highlight several critical findings for building robust multilingual voice assistants.
Encoder Performance and Translation Baselines
LaBSE consistently outperformed mUSE across nearly all configurations, establishing itself as the superior encoder for cross-lingual intent detection. Interestingly, the "ASR-then-translate" pipeline proved to be a formidable baseline. For high-resource languages, transcribing speech and translating it to English for an English-trained classifier often yielded results comparable to training on native data.
The Importance of In-Domain Data
A significant performance gap was observed between zero-shot and few-shot learning. While models could perform zero-shot transfer (training only on auxiliary English data), the average accuracy was only 72.0%. Introducing even a small amount of in-domain native data (few-shot) boosted average accuracy to over 95% in the "translate-to-EN" setup.
Robustness and Synergy
The study also found that training on all languages simultaneously (the "multilingual" setup) achieved a high average accuracy of 93.3%, suggesting that models can learn a shared intent space effectively. Furthermore, techniques like using $n$-best ASR hypotheses or multiple translation engines provided modest performance gains by acting as a form of data augmentation.
Conclusion
This work demonstrates that state-of-the-art multilingual encoders, combined with translation pipelines, can achieve high accuracy in intent detection from spoken data across diverse languages. However, the reliance on mature ASR and MT systems suggests that performance may vary for lower-resource languages. Future research should focus on extending these methodologies to underserved languages and more complex, fine-grained intent spaces.
Summary updated