Abstract
We propose a novel inference-time out-of-domain (OOD) detection algorithm for specialized large language models (LLMs). Despite achieving state-of-the-art performance on in-domain tasks through fine-tuning, specialized LLMs remain vulnerable to incorrect or unreliable outputs when presented with OOD inputs, posing risks in critical applications. Our method leverages the Inductive Conformal Anomaly Detection (ICAD) framework, using a new non-conformity measure based on the model’s dropout tolerance. Motivated by recent findings on polysemanticity and redundancy in LLMs, we hypothesize that in-domain inputs exhibit higher dropout tolerance than OOD inputs. We aggregate dropout tolerance across multiple layers via a valid ensemble approach, improving detection while maintaining theoretical false alarm bounds from ICAD. Experiments with medical-specialized LLMs show that our approach detects OOD inputs better than baseline methods, with AUROC improvements of 2% to 37% when treating OOD datapoints as positives and in-domain test datapoints as negatives.
Introduction
Specialized large language models (LLMs) are increasingly deployed in high-stakes domains such as medicine and law. While fine-tuning enables these models to achieve state-of-the-art performance on in-distribution (iD) tasks, they remain inherently brittle when encountering out-of-distribution (OOD) inputs. In such cases, a specialized model may confidently generate hallucinations or attempt to force an OOD query into its specialized context—for instance, a mental health LLM might interpret a physical injury through a psychological lens. Detecting these unreliable inputs at inference time is critical for safety, yet existing methods often lack theoretical guarantees or suffer from high false-alarm rates.
Methodology
We propose a novel inference-time OOD detection algorithm based on the concept of Dropout Tolerance. This approach is motivated by the hypothesis that specialized LLMs develop high redundancy (polysemanticity) for their specific domain. Consequently, iD queries should be robust to small network perturbations, whereas OOD queries—which the model is less equipped to process—should exhibit higher sensitivity.
The algorithm measures the "dropout tolerance" of a query by iteratively deactivating the most active neurons in a specific layer until the model's output changes semantically. This semantic shift is determined by an evaluator LLM (e.g., GPT-4o-mini). This tolerance score serves as a Non-Conformity Measure (NCM) within the Inductive Conformal Anomaly Detection (ICAD) framework. By comparing a query's tolerance against a calibration set of iD data, we derive a p-value. Queries with a p-value below a threshold $\epsilon$ are flagged as OOD. To enhance robustness, we ensemble scores across multiple layers (e.g., layers 7, 15, and 22) using valid averaging functions that preserve theoretical false-alarm bounds.
Experimental Setup
The method was evaluated using two specialized models: EYE-LLaMA (ophthalmology) and MentaLLaMA (mental health). In-domain performance was established using the EyeQA and IMHI datasets. OOD detection was tested against CovidQA, MedMCQA, and cross-domain queries (e.g., using EyeQA as OOD for MentaLLaMA).
Results and Discussion
Our approach significantly outperformed baseline methods, achieving AUROC scores between 0.83 and 0.96. Compared to simple base-score methods, we observed AUROC improvements ranging from 2% to 37%. Crucially, the empirical false alarm rate closely aligned with the theoretical threshold $\epsilon$, demonstrating the reliability of the conformal framework.
Analysis revealed that earlier layers are generally more sensitive to dropout, suggesting that initial query comprehension is more fragile than the generation phase. Additionally, multiple-choice questions were found to be more susceptible to dropout-induced changes than subjective, long-form responses.
Conclusion
While effective, the method introduces computational overhead due to multiple forward passes, though these can be parallelized. Future work will explore extending this framework to multi-modal LLMs and optimizing neuron selection for increased efficiency. By providing mathematically grounded OOD detection, Polysemantic Dropout offers a path toward safer deployment of specialized AI systems.
Summary updated