TL;DR
This paper introduces a lightweight, embedding-based method for detecting out-of-distribution (OOD) inputs in conditional language models (CLMs) like those used for summarization and translation. By combining these OOD scores with model perplexity, the authors enable "selective generation"—allowing models to safely abstain from producing low-quality or hallucinated outputs when faced with unfamiliar data.
What Problem Does This Paper Address?
Modern generative language models are incredibly capable but notoriously brittle when they encounter data that differs significantly from their training set. In these out-of-distribution (OOD) scenarios, models often become "confidently wrong," producing fluent-sounding but factually incorrect or nonsensical text.
For conditional language models (CLMs), this problem is compounded by their auto-regressive nature: a single mistake at the beginning of a sentence can propagate and magnify through every subsequent token. Furthermore, standard confidence metrics like perplexity (which measures how "surprised" a model is by a sequence) are surprisingly poor at identifying OOD inputs. As shown in the paper, the perplexity distributions for in-domain and OOD data often overlap so much that they are indistinguishable, making it impossible to set a reliable threshold for safety.
Methods
The authors propose a "lightweight" approach that leverages the internal representations already computed by the model during inference. Instead of looking at the final output probabilities (like perplexity does), they look at the embeddings (hidden states) from the encoder and decoder.
Key techniques include:
- Mahalanobis Distance (MD): They fit a Gaussian distribution to the embeddings of the training data. For a new input, they calculate its distance from this "known" distribution in the embedding space.
- Relative Mahalanobis Distance (RMD): To better handle "near-OOD" cases (data that is similar but not identical to training), they introduce a background model. They calculate the distance to the training distribution and subtract the distance to a "background" distribution fitted on a massive, diverse dataset (like C4). This highlights the features that make the input specifically OOD relative to the training task.
- Binary Logits Score: They also trained a simple logistic regression classifier to distinguish between training embeddings and background embeddings, using the resulting logits as an OOD score.
- Selective Generation: To decide whether to trust a generated output, they combine the OOD score (which captures epistemic uncertainty—what the model doesn't know) with perplexity (which captures aleatoric uncertainty—inherent noise or ambiguity).
Datasets
The researchers evaluated their methods on two primary tasks:
- Abstractive Summarization:
- In-domain: XSum (BBC news articles).
- Near-OOD: CNN/DailyMail and Newsroom (other news sources).
- Far-OOD: Reddit TIFU (informal stories), SAMSum, and ForumSum (dialogues).
- Background: C4 (Common Crawl).
- Machine Translation:
- In-domain: WMT15 English-French.
- OOD: Various OPUS subsets (Medical, Law, Koran, IT, Subtitles) and MTNT (noisy Reddit comments).
- Background: ParaCrawl.
Key Insights and Ideas
- Embeddings are more informative than probabilities: Softmax-based scores like perplexity are often "blind" to distribution shifts because the model's final layer is trained to be decisive. The internal hidden states retain much more information about how "unfamiliar" an input truly is.
- RMD provides a continuous spectrum: While binary classifiers are good at detection, the RMD score is better at distinguishing the degree of shift. It can clearly separate "near-OOD" from "far-OOD" data, which is crucial for nuanced deployment decisions.
- Aleatoric vs. Epistemic Uncertainty: The paper highlights that a model can be uncertain because the task is hard (aleatoric) or because it hasn't seen the data before (epistemic). A robust system must account for both. Combining perplexity with embedding-based distance provides a more comprehensive "trust score."
Key Results
- Superior Detection: The proposed RMD and Binary Logits scores achieved AUROCs (Area Under the ROC Curve) often exceeding 0.95 for summarization and 0.85-0.95 for translation. In contrast, the perplexity baseline often performed near 0.5, which is no better than random guessing.
- Effective Selective Generation: By using the combined score to decide when to abstain, the authors showed they could significantly improve the average quality of the remaining generated outputs. For example, in summarization, the combined score outperformed expensive external model-based baselines like NLI-based factual consistency scores.
- Efficiency: Because these scores are derived from existing model embeddings, they add negligible computational overhead compared to ensemble methods that require running the model multiple times.
Limitations and Future Work
- Architecture Scope: The study focused on encoder-decoder Transformers (like PEGASUS and T5). While the authors expect the findings to translate to decoder-only models (like GPT-4), this was not explicitly tested.
- Interpretability: While the system can flag that an input is OOD, it doesn't inherently explain why. The authors touched on sentence-level attribution (identifying which part of a text is most OOD), but more work is needed to make these signals human-interpretable.
- Background Data Sensitivity: The method relies on having a good "background" dataset. While results were robust across C4 and ParaCrawl, the choice of background data remains a hyperparameter that might need tuning for specialized domains.
Summary updated