Out-of-Distribution Detection and Selective Generation for Conditional Language Models

Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, Peter J. Liu

TL;DR

This paper introduces a lightweight, embedding-based method for detecting out-of-distribution (OOD) inputs in conditional language models (CLMs) like those used for summarization and translation. By combining these OOD scores with model perplexity, the authors enable "selective generation"—allowing models to safely abstain from producing low-quality or hallucinated outputs when faced with unfamiliar data.

What Problem Does This Paper Address?

Modern generative language models are incredibly capable but notoriously brittle when they encounter data that differs significantly from their training set. In these out-of-distribution (OOD) scenarios, models often become "confidently wrong," producing fluent-sounding but factually incorrect or nonsensical text.

For conditional language models (CLMs), this problem is compounded by their auto-regressive nature: a single mistake at the beginning of a sentence can propagate and magnify through every subsequent token. Furthermore, standard confidence metrics like perplexity (which measures how "surprised" a model is by a sequence) are surprisingly poor at identifying OOD inputs. As shown in the paper, the perplexity distributions for in-domain and OOD data often overlap so much that they are indistinguishable, making it impossible to set a reliable threshold for safety.

Figure 1: Perplexity scores density of a CLM trained on (a) xsum for summarization, and (b) WMT for translation, evaluated on other datasets/domains. Perplexity is not well suited for OOD detection due to significant overlap between in-domain and OOD scores.

Methods

The authors propose a "lightweight" approach that leverages the internal representations already computed by the model during inference. Instead of looking at the final output probabilities (like perplexity does), they look at the embeddings (hidden states) from the encoder and decoder.

Key techniques include:

  • Mahalanobis Distance (MD): They fit a Gaussian distribution to the embeddings of the training data. For a new input, they calculate its distance from this "known" distribution in the embedding space.
  • Relative Mahalanobis Distance (RMD): To better handle "near-OOD" cases (data that is similar but not identical to training), they introduce a background model. They calculate the distance to the training distribution and subtract the distance to a "background" distribution fitted on a massive, diverse dataset (like C4). This highlights the features that make the input specifically OOD relative to the training task.
  • Binary Logits Score: They also trained a simple logistic regression classifier to distinguish between training embeddings and background embeddings, using the resulting logits as an OOD score.
  • Selective Generation: To decide whether to trust a generated output, they combine the OOD score (which captures epistemic uncertainty—what the model doesn't know) with perplexity (which captures aleatoric uncertainty—inherent noise or ambiguity).
Figure 2: The proposed OOD detector based on input and output embeddings.

Datasets

The researchers evaluated their methods on two primary tasks:

  • Abstractive Summarization:
    • In-domain: XSum (BBC news articles).
  • Near-OOD: CNN/DailyMail and Newsroom (other news sources).
  • Far-OOD: Reddit TIFU (informal stories), SAMSum, and ForumSum (dialogues).
  • Background: C4 (Common Crawl).
  • Machine Translation:
    • In-domain: WMT15 English-French.
  • OOD: Various OPUS subsets (Medical, Law, Koran, IT, Subtitles) and MTNT (noisy Reddit comments).
  • Background: ParaCrawl.

Key Insights and Ideas

  • Embeddings are more informative than probabilities: Softmax-based scores like perplexity are often "blind" to distribution shifts because the model's final layer is trained to be decisive. The internal hidden states retain much more information about how "unfamiliar" an input truly is.
  • RMD provides a continuous spectrum: While binary classifiers are good at detection, the RMD score is better at distinguishing the degree of shift. It can clearly separate "near-OOD" from "far-OOD" data, which is crucial for nuanced deployment decisions.
  • Aleatoric vs. Epistemic Uncertainty: The paper highlights that a model can be uncertain because the task is hard (aleatoric) or because it hasn't seen the data before (epistemic). A robust system must account for both. Combining perplexity with embedding-based distance provides a more comprehensive "trust score."
Figure 3: Density of RMD (left) and Binary logits (right) OOD scores evaluated on summarization datasets. RMD is better at distinguishing near-OOD from far-OOD.

Key Results

  • Superior Detection: The proposed RMD and Binary Logits scores achieved AUROCs (Area Under the ROC Curve) often exceeding 0.95 for summarization and 0.85-0.95 for translation. In contrast, the perplexity baseline often performed near 0.5, which is no better than random guessing.
  • Effective Selective Generation: By using the combined score to decide when to abstain, the authors showed they could significantly improve the average quality of the remaining generated outputs. For example, in summarization, the combined score outperformed expensive external model-based baselines like NLI-based factual consistency scores.
  • Efficiency: Because these scores are derived from existing model embeddings, they add negligible computational overhead compared to ensemble methods that require running the model multiple times.

Limitations and Future Work

  • Architecture Scope: The study focused on encoder-decoder Transformers (like PEGASUS and T5). While the authors expect the findings to translate to decoder-only models (like GPT-4), this was not explicitly tested.
  • Interpretability: While the system can flag that an input is OOD, it doesn't inherently explain why. The authors touched on sentence-level attribution (identifying which part of a text is most OOD), but more work is needed to make these signals human-interpretable.
  • Background Data Sensitivity: The method relies on having a good "background" dataset. While results were robust across C4 and ParaCrawl, the choice of background data remains a hyperparameter that might need tuning for specialized domains.

Summary updated

Abstract

Machine learning algorithms typically assume independent and identically distributed samples in training and at test time. Much work has shown that high-performing ML classifiers can degrade significantly and provide overly-confident, wrong classification predictions, particularly for out-of-distribution (OOD) inputs. Conditional language models (CLMs) are predominantly trained to classify the next token in an output sequence, and may suffer even worse degradation on OOD inputs as the prediction is done auto-regressively over many steps. Furthermore, the space of potential low-quality outputs is larger as arbitrary text can be generated and it is important to know when to trust the generated output. We present a highly accurate and lightweight OOD detection method for CLMs, and demonstrate its effectiveness on abstractive summarization and translation. We also show how our method can be used under the common and realistic setting of distribution shift for selective generation (analogous to selective prediction for classification) of high-quality outputs, while automatically abstaining from low-quality ones, enabling safer deployment of generative language models.
BibTeX
@inproceedings{ren2023out,
  author = {Jie Ren and Jiaming Luo and Yao Zhao and Kundan Krishna and Mohammad Saleh and Balaji Lakshminarayanan and Peter J. Liu},
  title = {{Out-of-Distribution Detection and Selective Generation for Conditional Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=kJUS5nD0vPB},
}
Figure