Benchmarking Language-agnostic Intent Classification for Virtual Assistant Platforms

Gengyu Wang, Cheng Qian, Lin Pan, Haode Qi, Ladislav Kunc, Saloni PotdarNAACL 2022

TL;DR

This paper tackles the challenge of building virtual assistants (VAs) that can understand user intent across many languages without requiring extensive, language-specific engineering. They introduce a comprehensive benchmark to evaluate language-agnostic intent classification, finding that while large multilingual models achieve high accuracy, they are too slow for practical commercial use. IBM Watson Assistant emerges as a strong contender, offering a compelling balance of accuracy and speed.

What Problem Does This Paper Address?

Virtual assistants (VAs) are becoming ubiquitous, but supporting them across the vast diversity of human languages is a significant hurdle. Traditionally, each VA component (like tokenizers and intent classifiers) is meticulously crafted for specific languages. This makes adding support for a new language a costly and time-consuming process involving expensive re-training and redesign. The paper addresses the critical need for "language-agnostic" intent classification – a single system that can handle intent recognition across many languages without specialized engineering for each. This is particularly challenging for low-resource languages where data is scarce. Furthermore, in commercial VA platforms, models must not only be accurate but also efficient, with training times measured in seconds rather than hours, to enable rapid iteration and deployment.

Methods

The authors propose a benchmarking framework to evaluate language-agnostic intent classification. Their approach involves:

  • Comprehensive Evaluation: They assess seven different intent classification models/services, including two commercial VA platforms (IBM Watson Assistant and RASA) and five representative multilingual pre-trained language models (LMs).
  • Diverse Datasets: The benchmark utilizes a mix of 9 public datasets and 20 internal, real-world datasets, covering 10 low-resource languages and 9 domains. This ensures a robust evaluation across various linguistic and domain-specific challenges.
  • Multiple Settings: They evaluate models under three scenarios:
  • Standard Training: Training on the full dataset and testing on the full test set.
  • Few-Shot Training: Simulating real-world scenarios where data is scarce, by training on subsets of 5, 15, and 30 examples per intent class.
  • Difficult Test Sets: Creating specialized test sets (using TF*IDF and Jaccard similarity) to better differentiate model performance when standard test sets might be too similar to training data.
  • Focus on Trade-offs: Beyond accuracy, they meticulously measure training time and computational requirements, recognizing the critical importance of efficiency in commercial applications.

Datasets

The benchmark is built upon a substantial collection of datasets:

  • Public Datasets (5 languages):
    • MTOP (Hindi, Thai)
    • MultiATIS (Hindi, Turkish)
    • Leyzer (Polish)
    • Chatbot-ru (Russian)
    • PSTU (Russian)
  • Internal Datasets (7 languages, 9 domains): These are real-world datasets curated from a commercial VA platform, covering domains like banking, COVID-19, and telecommunication.
  • Difficult Test Sets: Created by selecting challenging examples from the original test sets based on TF*IDF and Jaccard similarity to better distinguish model performance.

Key Insights and Ideas

  • The "Big LM" Dilemma: While large, pre-trained multilingual LMs like LaBSE achieve the highest accuracy across most settings, their training times are prohibitively long for commercial deployment. This highlights a crucial trade-off between raw performance and practical usability.
  • Watson Assistant's Sweet Spot: IBM Watson Assistant demonstrates a remarkable balance. It achieves near-top accuracy while offering significantly faster training times, making it a strong candidate for real-world applications where rapid iteration is key.
  • Few-Shot Learning is Crucial: The performance drop observed in few-shot settings underscores the importance of models that can generalize well from limited data, a common scenario in enterprise VA development.
  • Difficult Test Sets Matter: The creation of specialized difficult test sets proved effective in revealing performance differences that might be masked by standard test splits, especially when models achieve high overall accuracy.

Key Results

  • Accuracy Leader: LaBSE consistently achieved the highest accuracy across most evaluation settings.
  • Best Trade-off: IBM Watson Assistant offered the best accuracy-training time trade-off, achieving competitive accuracy with training times often under a minute. For example, on the Leyzer dataset, WA trained in under a minute, while XLM-R took over 100 minutes.
  • Few-Shot Performance: In few-shot settings (5 examples per intent), WA showed an advantage over RASA and mBERT in average accuracy across languages.
  • Accuracy Drop in Few-Shot: Models experienced a significant drop in accuracy in few-shot settings compared to full training, decreasing from around 80% to 60% on average.

Limitations and Future Work

  • Limited Commercial VA Coverage: The authors acknowledge that they only benchmarked two commercial VA platforms (WA and RASA) due to terms of use restrictions from other providers. A broader evaluation would be beneficial.
  • Focus on Intent Classification: The study is specifically focused on intent classification, which is a critical component but only one part of a complete VA system. Future work could extend this benchmarking to other VA components.
  • Computational Resources: While they mention resource types (CPU/GPU), a more detailed analysis of computational resource requirements across all models and settings could provide further insights for practitioners.
  • Encouraging Further Research: The authors hope their work will spur more research and development in language-agnostic chatbot solutions, particularly for low-resource languages.

Summary updated

BibTeX
@inproceedings{wang2022benchmarking,
  author = {Gengyu Wang and Cheng Qian and Lin Pan and Haode Qi and Ladislav Kunc and Saloni Potdar},
  title = {{Benchmarking Language-agnostic Intent Classification for Virtual Assistant Platforms}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}

Loading related content...

Figure