Likelihood Ratios and Generative Classifiers for Unsupervised Out-of-Domain Detection in Task Oriented Dialog

Varun Gangal, Abhinav Arora, Arash Einolghozati, Sonal GuptaAAAI 2020 · DOI

Abstract

Task-oriented dialog systems often suffer from over-confidence when encountering out-of-domain (OOD) inputs, leading to confident misclassifications. This paper introduces ROSTD, a realistic dataset of human-authored OOD sentences, and proposes a detection method using generative classifiers combined with likelihood ratios. By modeling the joint probability of inputs and labels and utilizing a background model to filter generic linguistic features, the proposed method significantly outperforms standard discriminative baselines, achieving an AUROC of 98.22 on the ROSTD benchmark.

Introduction

When a user interacts with a task-oriented bot, they may provide inputs that fall outside the system's supported intents. For instance, asking a weather bot "How do I fix a pot of coffee?" should ideally trigger an "out-of-domain" response. However, standard discriminative models—which learn boundaries between known classes rather than modeling the data distribution itself—often assign high confidence scores to such unrelated inputs. This "over-confidence" problem is exacerbated by the use of synthetic OOD datasets in research, which fail to capture the complexity of real-world user errors.

Methods

The authors propose a shift from discriminative to generative modeling to improve OOD detection.

Generative Classifiers ($L_{gen}$)

Instead of modeling $P(y|x)$, the authors use a generative approach to model the joint probability $P(x, y)$. This allows the system to compute the marginal likelihood $P(x)$, providing a measure of how likely a given sentence is to appear in the training distribution.

Architecture of the generative classifier modeling joint probability P(x, y).

Likelihood Ratios (LLR) and Background Models

Raw likelihood $P(x)$ can be misleading because common linguistic features (e.g., stop words) may make OOD sentences appear likely. To address this, the authors introduce a Background Model trained on "noised" versions of the training data. The final OOD score is calculated as the Likelihood Ratio: $$\text{LLR}(x) = \log P_{\text{model}}(x) - \log P_{\text{background}}(x)$$ This ratio effectively cancels out generic linguistic noise, highlighting task-specific features. For NLP, the authors found that a "Uniroot" noising strategy—sampling word substitutions based on the square root of frequency—provided the best balance for the background model.

Datasets

The study utilizes two primary datasets:

  1. SNIPS: A standard public dataset used for synthetic OOD benchmarks.
  2. ROSTD (Real Out-of-Domain Sentences From Task-oriented Dialog): A new contribution featuring ~4,500 human-authored OOD examples. Unlike synthetic sets, ROSTD includes nuanced categories like underspecified queries and subjective questions, providing a more rigorous test for OOD detection.

Key Results

The proposed LLR method demonstrates substantial improvements over traditional baselines, particularly on the realistic ROSTD dataset.

Comparison of score distributions for In-Domain (ID) vs. Out-of-Domain (OOD) data using raw likelihood (a) vs. the proposed Likelihood Ratio (b).

As shown in the histograms, the raw generative likelihood ($L_{gen}$) shows significant overlap between ID and OOD samples. However, the addition of the Background Model and Likelihood Ratio ($L_{gen} + \text{BACKLM} + \text{UNIROOT}$) creates a clear separation.

  • AUROC Gains: On ROSTD, the AUROC improved from 70.75 (Maximum Softmax Probability baseline) to 98.22 with the LLR method.
  • Error Reduction: The False Positive Rate at 95% recall (FPR@95%TPR) dropped from 100% in the baseline to just 7.41%, indicating a massive reduction in confident misclassifications.

Conclusion

This work highlights that likelihood alone is insufficient for robust OOD detection in NLP. By employing generative classifiers and likelihood ratios with a background model, systems can better distinguish between task-relevant inputs and irrelevant noise. Future work will explore more complex noising functions and the application of these techniques to multi-turn dialog environments.

Summary updated

BibTeX
@inproceedings{gangal2020likelihood,
  author = {Varun Gangal and Abhinav Arora and Arash Einolghozati and Sonal Gupta},
  title = {{Likelihood Ratios and Generative Classifiers for Unsupervised Out-of-Domain Detection in Task Oriented Dialog}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6280},
}
Figure