A Scalable Neural Shortlisting-Reranking Approach for Large-Scale Domain Classification in Natural Language Understanding

Young-Bum Kim, Dongchan Kim, Joo-Kyung Kim, Ruhi Sarikaya

TL;DR

This paper introduces a two-stage neural architecture—Shortlisting and Reranking—to solve the massive scale problem of domain classification in digital assistants like Alexa. By first using a lightweight model to prune thousands of domains down to a "k-best" list and then applying a sophisticated list-wise BiLSTM reranker with rich contextual features, the authors achieve high accuracy and low latency on a dataset of 1,500 overlapping domains.

What Problem Does This Paper Address?

Modern Intelligent Personal Digital Assistants (IPDAs) are evolving from handling a few dozen domains (like Weather or Music) to thousands of independent, often overlapping, third-party skills. This creates two major hurdles:

  1. Ambiguity: An utterance like "play michael jackson" could be handled by a music streaming domain, a video domain, or a radio domain. Without a shared schema, finding the "best" one is difficult.
  2. Scalability: Running full Natural Language Understanding (NLU) pipelines—including intent classification and slot filling—across thousands of domains in real-time is computationally impossible due to latency and memory constraints.

Traditional systems often force users to use "stopgap" phrases (e.g., "Ask AllRecipes how to bake a pie"), which feels unnatural. This paper seeks a way to handle natural, ambiguous queries at scale.

Methods

The authors propose a two-step pipeline designed for maximal efficiency:

1. The Shortlister

The first stage is a lightweight BiLSTM model that uses only character and word-level embeddings. Its goal is high recall: quickly trimming the search space from 1,500 domains to a small "k-best" list (e.g., top 5).

  • Orthography-sensitive embeddings: By looking at character-level patterns, the model handles out-of-vocabulary words better.
  • Softmax Strategies: The authors compared a standard softmax ($softmax_a$) against a per-domain binary classification ($softmax_b$). They found $softmax_b$ more effective for large-scale settings because it doesn't unfairly suppress overlapping domains.
Figure 2: The architecture of our neural shortlisting model that uses character and word-level information of a given ut- terance.

2. Hypotheses Reranker (HypRank)

The second stage takes the k-best domains and performs a list-wise reranking. Unlike previous "point-wise" approaches that look at each domain in isolation, HypRank uses a BiLSTM to look at the entire list of candidates simultaneously, capturing cross-hypothesis features automatically.

  • Rich Context: It incorporates NLU signals (intent scores, slot tagging confidence), user preferences (which domains the user has enabled), and domain popularity.
  • Pre-trained Embeddings: It uses pre-trained embeddings for domains, intents, and slots to provide a dense semantic representation of the hypotheses.
Architecture of the Hypotheses Reranker

Datasets

The models were evaluated on two distinct settings:

  • Traditional IPDA: A simulated set of ~20 non-overlapping, high-quality built-in domains with 4 million utterances.
  • Large-Scale IPDA: A massive set of 1,500 domains (categorized into Device, Food, Ent., Info., News, Shopping, and Util.) with over 6 million utterances. This represents the messy, overlapping reality of modern voice assistants.

Key Insights and Ideas

  • The Power of Characters: Even without complex context, character and word-level BiLSTMs are surprisingly discriminative for initial shortlisting.
  • List-wise > Point-wise: The BiLSTM reranker outperforms traditional logistic regression or point-wise neural networks because it can "see" the competition. It learns, for example, that if one domain has a very high slot-tagging confidence compared to the others, it is likely the winner.
  • Context is King for Disambiguation: In the large-scale setting, the gap between the Shortlister (81.49% accuracy) and the Reranker (93.83% accuracy) is huge. This proves that for thousands of domains, surface-level text isn't enough; you need user history and domain popularity to break ties.

Key Results

The two-step approach significantly outperformed baselines in the large-scale 1,500-domain environment.

Figure 1: A high-level flow of our two-step shortlisting-reranking approach given an utterance to an IPDA.
  • Shortlister Performance: Achieving a 5-best accuracy of 95.93% on 1,500 domains, providing a strong foundation for the reranker.
  • Reranker Accuracy: The best-performing model ($LSTMC$, which concatenates hypothesis vectors with BiLSTM outputs) reached 93.83% accuracy. This is a massive improvement over the 1-best Shortlister result of 81.49%.
  • Efficiency: By only running full NLU analysis on the top 5 candidates, the system maintains the low latency required for real-time production environments.

Limitations and Future Work

  • First-Turn Only: The current scope is limited to the first utterance in a conversation. The authors note that multi-turn contextual signals are a logical next step.
  • Shortlister Context: Currently, the shortlister doesn't use user preferences or popularity to save on computation. The authors suggest exploring ways to include minimal context in the first stage without blowing up latency.
  • Manual vs. Automatic Features: Interestingly, adding manually designed cross-hypothesis features to the neural reranker did not improve performance, suggesting the BiLSTM is already capturing those relationships effectively.

Summary updated

Abstract

Intelligent personal digital assistants (IPDAs), a popular real-life application with spoken language understanding capabilities, can cover potentially thousands of overlapping domains for natural language understanding, and the task of finding the best domain to handle an utterance becomes a challenging problem on a large scale. In this paper, we propose a set of efficient and scalable neural shortlisting-reranking models for large-scale domain classification in IPDAs. The shortlisting stage focuses on efficiently trimming all domains down to a list of k-best candidate domains, and the reranking stage performs a list-wise reranking of the initial k-best domains with additional contextual information. We show the effectiveness of our approach with extensive experiments on 1,500 IPDA domains.
BibTeX
@inproceedings{kim2018scalable,
  author = {Young-Bum Kim and Dongchan Kim and Joo-Kyung Kim and Ruhi Sarikaya},
  title = {{A Scalable Neural Shortlisting-Reranking Approach for Large-Scale Domain Classification in Natural Language Understanding}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=HyEA9X-O-B},
}
Figure