Polysemantic Dropout: Conformal OOD Detection for Specialized LLMs

Ayush Gupta, Ramneet Kaur, Anirban Roy, Adam D. Cobb, Rama Chellappa, Susmit Jha

Abstract

We propose a novel inference-time out-of-domain (OOD) detection algorithm for specialized large language models (LLMs). Despite achieving state-of-the-art performance on in-domain tasks through fine-tuning, specialized LLMs remain vulnerable to incorrect or unreliable outputs when presented with OOD inputs, posing risks in critical applications. Our method leverages the Inductive Conformal Anomaly Detection (ICAD) framework, using a new non-conformity measure based on the model's dropout tolerance. Motivated by recent findings on polysemanticity and redundancy in LLMs, we hypothesize that in-domain inputs exhibit higher dropout tolerance than OOD inputs. We aggregate dropout tolerance across multiple layers via a valid ensemble approach, improving detection while maintaining theoretical false alarm bounds from ICAD. Experiments with medical-specialized LLMs show that our approach detects OOD inputs better than baseline methods, with AUROC improvements of $2\%$ to $37\%$ when treating OOD datapoints as positives and in-domain test datapoints as negatives.

Introduction

Fine-tuning Large Language Models (LLMs) for specialized domains—such as ophthalmology or mental health—has yielded impressive results. However, these specialized models often suffer from a dangerous "over-confidence" problem. When presented with Out-of-Distribution (OOD) queries that fall outside their specific expertise, they rarely signal uncertainty. Instead, they tend to hallucinate or force-fit the query into their specialized context. For example, a model trained specifically for mental health might incorrectly interpret a general medical triage query through a psychiatric lens.

In safety-critical fields like clinical decision support, these "silent failures" are unacceptable. To address this, we introduce a model-agnostic, inference-time method to detect OOD inputs without requiring expensive retraining or access to the original training weights.

Methodology

Polysemantic Dropout and Dropout Tolerance

Our approach, Polysemantic Dropout, is built on the Inductive Conformal Anomaly Detection (ICAD) framework. The core metric is "dropout tolerance": the minimum fraction of neurons in a specific layer that must be deactivated before the model's output changes semantically.

Our hypothesis is rooted in the concept of polysemanticity—the phenomenon where neurons activate on multiple related concepts. We posit that in-domain (ID) inputs benefit from higher redundancy within the model's weights. Consequently, an ID query should be robust to internal noise (dropout), whereas an OOD query—which the model has not learned to represent efficiently—should be "brittle," with its output changing significantly even at low dropout rates.

Non-Conformity Measure (NCM)

We define the Non-Conformity Measure (NCM) as $1 - \text{dropout tolerance}$. A higher NCM score indicates a higher likelihood that the input is OOD. To determine if a response has "changed" following dropout, we utilize a secondary LLM (GPT-4o) to evaluate the semantic similarity between the original and the perturbed outputs.

Layer Ensembling

Rather than relying on a single layer, our algorithm aggregates p-values from multiple layers (e.g., layers 7, 15, and 22) using valid merging functions such as the Arithmetic or Geometric Mean. This ensemble approach improves detection sensitivity while preserving the theoretical false alarm bounds provided by the ICAD framework.

Experimental Setup

Models and Datasets

We evaluated our method using two specialized models: EYE-LLaMA (ophthalmology) and MentaLLaMA (mental health).

  • In-Distribution (ID): We used EyeQA (~18,000 pairs) and IMHI (~105,000 pairs) for the respective domains.
  • Out-of-Distribution (OOD): We utilized COVID-QA (scientific articles on COVID-19) and MedMCQA (general medical entrance exams, filtered to exclude ophthalmology and mental health topics).

Results and Discussion

Performance Metrics

The proposed method consistently outperformed standard OOD detection baselines. We observed AUROC (Area Under the Receiver Operating Characteristic) improvements ranging from 2% to 37%. Specifically, the ensemble approach achieved an AUROC of 0.95-0.96 for MentaLLaMA and 0.83-0.91 for EYE-LLaMA.

Key Insights

  • Redundancy as a Safety Signal: Specialized training creates redundant pathways for domain-specific knowledge. This redundancy makes the model robust to internal noise for familiar topics, a robustness that vanishes when processing unfamiliar OOD data.
  • Layer Sensitivity: Earlier layers (e.g., layer 7) appear more critical for initial query understanding. Dropout in these layers changes responses more frequently than in later layers, making them highly sensitive indicators for OOD detection.
  • Question Format: Multiple-choice questions (MCQs) are significantly more brittle than long-form, subjective questions. A minor shift in internal logic can easily flip a single-letter choice (e.g., A to B), whereas the semantic core of a paragraph often remains stable despite internal noise.
  • Reliable Bounds: The algorithm empirically respects the user-defined "false alarm" limits. This adherence to theoretical bounds is crucial for establishing trust in clinical and high-stakes environments.

Conclusion

Polysemantic Dropout provides a robust, theoretically grounded method for OOD detection in specialized LLMs. While the method introduces some computational overhead due to multiple forward passes and the need for a secondary "judge" LLM, it significantly enhances the safety profile of specialized AI. Future work will focus on extending this framework to multi-modal models and developing more efficient methods for measuring semantic change.

Summary updated

Abstract

We propose a novel inference-time out-of-domain (OOD) detection algorithm for specialized large language models (LLMs). Despite achieving state-of-the-art performance on in-domain tasks through fine-tuning, specialized LLMs remain vulnerable to incorrect or unreliable outputs when presented with OOD inputs, posing risks in critical applications. Our method leverages the Inductive Conformal Anomaly Detection (ICAD) framework, using a new non-conformity measure based on the model's dropout tolerance. Motivated by recent findings on polysemanticity and redundancy in LLMs, we hypothesize that in-domain inputs exhibit higher dropout tolerance than OOD inputs. We aggregate dropout tolerance across multiple layers via a valid ensemble approach, improving detection while maintaining theoretical false alarm bounds from ICAD. Experiments with medical-specialized LLMs show that our approach detects OOD inputs better than baseline methods, with AUROC improvements of $2\%$ to $37\%$ when treating OOD datapoints as positives and in-domain test datapoints as negatives.
BibTeX
@inproceedings{gupta2025polysemantic,
  author = {Ayush Gupta and Ramneet Kaur and Anirban Roy and Adam D. Cobb and Rama Chellappa and Susmit Jha},
  title = {{Polysemantic Dropout: Conformal OOD Detection for Specialized LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=P0VyIuJ7AW},
}
Figure