Intent Detection with WikiHow

Li Zhang, Qing Lyu, Chris Callison-BurcharXiv 2020 · arXiv

TL;DR

Pretraining language models on wikiHow articles—by mapping instructional steps to overall goals—creates highly robust intent detection systems. This simple yet clever data augmentation strategy achieves state-of-the-art performance across multiple benchmarks and drastically improves accuracy in low-resource, few-shot scenarios.

What Problem Does This Paper Address?

Task-oriented dialog systems (like Apple's Siri or Amazon Alexa) rely on intent detection to understand what a user wants and map their natural language input to a specific action or API. For instance, mapping "wake me up at 8" to the intent Set an Alarm.

While modern neural networks handle standard intent detection well, they struggle when moving to new domains or new languages because annotated training data is scarce and expensive to produce. While most research focuses on engineering complex model architectures to solve this, this paper addresses the data bottleneck directly by leveraging massive, unstructured online instructional resources.

Methods

The authors introduce a novel pretraining task derived entirely from wikiHow, a website containing over 110,000 professionally edited instructional articles. They noticed a natural parallel between wikiHow's structure and dialog systems:

  • The Step (Utterance): The bolded header of a step in an article (e.g., "find good gas prices").
  • The Goal (Intent): The title of the article (e.g., "How to Save Money on Gas").

They framed the pretraining as a 4-choose-1 multiple-choice task. Given a specific step, the model must infer the correct goal out of four candidates. To prevent the task from being too trivial, negative candidates are sampled recursively from the "related articles" section to ensure they are semantically challenging but distinct.

For the models, they fine-tuned RoBERTa-large for English datasets and XLM-RoBERTa-base for multilingual datasets. During evaluation, standard intent detection datasets were also cast into this multiple-choice format, where the user utterance is the input and the full set of possible dataset intents serves as the candidates.

Datasets

  • wikiHow Pretraining Dataset (New): Automatically constructed from wikiHow crawls in English (107,298 examples), Spanish (64,803 examples), and Thai (6,342 examples).
  • The Snips Dataset: A popular single-turn English personal assistant benchmark with 7 intents.
  • The Schema-Guided Dialogue (SGD) Dataset: The largest multi-turn English dialog corpus to date, spanning dozens of domains.
  • Facebook Multilingual Datasets (FB-en/es/th): A single-turn dataset containing utterances annotated with intents and slots across English, Spanish, and Thai.

Key Insights and Ideas

  • Instructional Data Inherently Mirrors Intent: Human behavior is goal-oriented, and the way we document instructions online naturally maps actions to intents. This makes wikiHow a goldmine for training goal-oriented AI.
  • Cross-Lingual Transfer Power: For languages with smaller wikiHow corpora (like Thai), pretraining on the massive English wikiHow dataset and transferring cross-lingually yielded better results than training on the native-language wikiHow data.
  • Fixing the Low-Resource Fragility: Standard transformer models perform terribly when training data is scarce. By using wikiHow as a stepping stone, models develop a robust common-sense understanding of goals before they ever see a domain-specific dataset.

Key Results

The wikiHow-pretrained models achieved new state-of-the-art (SOTA) accuracy across all evaluated benchmarks:

  • Snips: 99.4%
  • SGD: 95.1%
  • Facebook Multilingual: 99.5% (English), 99.0% (Spanish), 97.6% (Thai)

The most dramatic impact was seen in low-resource settings. When restricted to just 100 in-domain training examples, vanilla transformer models collapsed to under 50% accuracy. In contrast, the wikiHow-pretrained models maintained over 75% accuracy across the board.

Figure 1: Learning curves of models in low-resource settings. The vertical axis is the accuracy of intent detection, while the horizontal axis is the number of in-domain training examples of each task, distorted to log-scale.

Limitations and Future Work

The authors point out a surprising problem: standard intent detection benchmarks are essentially "solved," with models hitting over 99% accuracy. Manual error analysis revealed that the remaining mistakes were mostly due to ambiguous or flat-out incorrect ground-truth labels in the benchmarks themselves.

However, these benchmarks only feature a dozen intents. Real-world assistants need to support thousands. To prove their model scales, the authors ran a preliminary experiment holding out 5,000 wikiHow goals as an "open-domain" intent space. While the model achieved a promising 36% top-1 accuracy out of 100,000 possible intents, the authors emphasize that open-domain intent detection is the true frontier for future research.

Summary updated

Abstract

Modern task-oriented dialog systems need to reliably understand users' intents. Intent detection is most challenging when moving to new domains or new languages, since there is little annotated data. To address this challenge, we present a suite of pretrained intent detection models. Our models are able to predict a broad range of intended goals from many actions because they are trained on wikiHow, a comprehensive instructional website. Our models achieve state-of-the-art results on the Snips dataset, the Schema-Guided Dialogue dataset, and all 3 languages of the Facebook multilingual dialog datasets. Our models also demonstrate strong zero- and few-shot performance, reaching over 75% accuracy using only 100 training examples in all datasets.
BibTeX
@misc{zhang2020intent,
  author = {Li Zhang and Qing Lyu and Chris Callison-Burch},
  title = {{Intent Detection with WikiHow}},
  year = {2020},
  eprint = {2009.05781},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.05781},
}
Figure