Polysemantic Dropout: Conformal OOD Detection for Specialized LLMs

Ayush Gupta, Ramneet Kaur, Anirban Roy, Adam D. Cobb, Rama Chellappa, Susmit Jha

Abstract

We propose a novel inference-time out-of-domain (OOD) detection algorithm for specialized large language models (LLMs). Despite achieving state-of-the-art performance on in-domain tasks through fine-tuning, specialized LLMs remain vulnerable to incorrect or unreliable outputs when presented with OOD inputs, posing risks in critical applications. Our method leverages the Inductive Conformal Anomaly Detection (ICAD) framework, using a new non-conformity measure based on the model’s dropout tolerance. Motivated by recent findings on polysemanticity and redundancy in LLMs, we hypothesize that in-domain inputs exhibit higher dropout tolerance than OOD inputs. We aggregate dropout tolerance across multiple layers via a valid ensemble approach, improving detection while maintaining theoretical false alarm bounds from ICAD. Experiments with medical-specialized LLMs show that our approach detects OOD inputs better than baseline methods, with AUROC improvements of 2% to 37% when treating OOD datapoints as positives and in-domain test datapoints as negatives.

Introduction

Specialized large language models (LLMs) are increasingly deployed in high-stakes domains such as medicine and law. While fine-tuning enables these models to achieve state-of-the-art performance on in-distribution (iD) tasks, they remain inherently brittle when encountering out-of-distribution (OOD) inputs. In such cases, a specialized model may confidently generate hallucinations or attempt to force an OOD query into its specialized context—for instance, a mental health LLM might interpret a physical injury through a psychological lens. Detecting these unreliable inputs at inference time is critical for safety, yet existing methods often lack theoretical guarantees or suffer from high false-alarm rates.

Methodology

We propose a novel inference-time OOD detection algorithm based on the concept of Dropout Tolerance. This approach is motivated by the hypothesis that specialized LLMs develop high redundancy (polysemanticity) for their specific domain. Consequently, iD queries should be robust to small network perturbations, whereas OOD queries—which the model is less equipped to process—should exhibit higher sensitivity.

The algorithm measures the "dropout tolerance" of a query by iteratively deactivating the most active neurons in a specific layer until the model's output changes semantically. This semantic shift is determined by an evaluator LLM (e.g., GPT-4o-mini). This tolerance score serves as a Non-Conformity Measure (NCM) within the Inductive Conformal Anomaly Detection (ICAD) framework. By comparing a query's tolerance against a calibration set of iD data, we derive a p-value. Queries with a p-value below a threshold $\epsilon$ are flagged as OOD. To enhance robustness, we ensemble scores across multiple layers (e.g., layers 7, 15, and 22) using valid averaging functions that preserve theoretical false-alarm bounds.

Overview of the ICAD framework using Dropout Tolerance to detect OOD queries.

Experimental Setup

The method was evaluated using two specialized models: EYE-LLaMA (ophthalmology) and MentaLLaMA (mental health). In-domain performance was established using the EyeQA and IMHI datasets. OOD detection was tested against CovidQA, MedMCQA, and cross-domain queries (e.g., using EyeQA as OOD for MentaLLaMA).

Results and Discussion

Our approach significantly outperformed baseline methods, achieving AUROC scores between 0.83 and 0.96. Compared to simple base-score methods, we observed AUROC improvements ranging from 2% to 37%. Crucially, the empirical false alarm rate closely aligned with the theoretical threshold $\epsilon$, demonstrating the reliability of the conformal framework.

ROC curves showing the performance of the proposed ensemble method compared to single-layer baselines across different models and datasets.

Analysis revealed that earlier layers are generally more sensitive to dropout, suggesting that initial query comprehension is more fragile than the generation phase. Additionally, multiple-choice questions were found to be more susceptible to dropout-induced changes than subjective, long-form responses.

Conclusion

While effective, the method introduces computational overhead due to multiple forward passes, though these can be parallelized. Future work will explore extending this framework to multi-modal LLMs and optimizing neuron selection for increased efficiency. By providing mathematically grounded OOD detection, Polysemantic Dropout offers a path toward safer deployment of specialized AI systems.

Summary updated

Abstract

We propose a novel inference-time out-of-domain (OOD) detection algorithm for specialized large language models (LLMs). Despite achieving state-of-the-art performance on in-domain tasks through fine-tuning, specialized LLMs remain vulnerable to incorrect or unreliable outputs when presented with OOD inputs, posing risks in critical applications. Our method leverages the Inductive Conformal Anomaly Detection (ICAD) framework, using a new non-conformity measure based on the model’s dropout tolerance. Motivated by recent findings on polysemanticity and redundancy in LLMs, we hypothesize that in-domain inputs exhibit higher dropout tolerance than OOD inputs. We aggregate dropout tolerance across multiple layers via a valid ensemble approach, improving detection while maintaining theoretical false alarm bounds from ICAD. Experiments with medical-specialized LLMs show that our approach detects OOD inputs better than baseline methods, with AUROC improvements of 2% to 37% when treating OOD datapoints as positives and in-domain test datapoints as negatives.
BibTeX
@inproceedings{gupta2025polysemantic,
  author = {Ayush Gupta and Ramneet Kaur and Anirban Roy and Adam D. Cobb and Rama Chellappa and Susmit Jha},
  title = {{Polysemantic Dropout: Conformal OOD Detection for Specialized LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=P0VyIuJ7AW},
}
Figure