Abstract
Retrieval-Augmented Generation (RAG) systems are increasingly deployed in high-stakes domains, where safety depends not only on how a system answers, but also on whether a query should be answered given a knowledge base (KB). Out-of-domain (OOD) queries can cause dense retrieval to surface weakly related context and lead the generator to produce fluent but unjustified responses. We study lightweight, KB-aligned OOD detection as an always-on gate for RAG systems. Our approach applies PCA to KB embeddings and scores queries in a compact subspace selected either by explained-variance retention (EVR) or by a separability-driven t-test ranking. We evaluate geometric semantic-search rules and lightweight classifiers across 16 domains, including high-stakes COVID-19 and Substance Use KBs, and stress-test robustness using both LLM-generated attacks and an in-the-wild 4chan attack. We find that low-dimensional detectors achieve competitive OOD performance while being faster, cheaper, and more interpretable than prompted LLM-based judges. Finally, human and LLM-based evaluations show that OOD queries primarily degrade the relevance of RAG outputs, showing the need for efficient external OOD detection to maintain safe, in-scope behavior.
The "Forced Choice" Problem in RAG
Retrieval-Augmented Generation (RAG) systems are designed to ground AI responses in a specific set of documents. However, they suffer from a "forced choice" problem: if a user asks a question completely unrelated to the knowledge base, the retriever will still surface the "least-irrelevant" documents it can find. This often leads the generator to produce fluent but entirely unjustified or unsafe responses—a major risk in high-stakes fields like medicine or public health.
While developers often use "guardrails" or additional LLM calls to judge query relevance, these methods are computationally expensive, slow (adding seconds of latency), and act as "black boxes." There is a clear need for an "always-on" gate that is fast, cheap, and explains why a query was rejected.
Methodology: PCA-Based Subspace Selection
The authors propose a straightforward but effective geometric approach to OOD detection:
- Embedding & PCA: They take the embeddings of the Knowledge Base (KB) and apply Principal Component Analysis (PCA) to identify the "shape" of the in-domain data.
- Subspace Selection: Instead of just taking the top components, they compare two strategies:
- Explained Variance Retention (EVR): Keeping components with the most data spread (standard PCA).
- p-value Ranking: Using a t-test to identify which specific principal components best separate in-domain queries from OOD queries.
- Low-Dimensional Projection: Queries are projected into a very compact subspace (often < 15 dimensions).
- Geometric Classifiers: They test simple rules like ε-ball (is the query within a certain radius of KB samples?) and ε-rect, alongside lightweight machine learning models like Gaussian Mixture Models (GMMs) and SVMs.
Experimental Setup and Datasets
The study evaluates the method across 16 domains, with a focus on high-stakes environments:
- COVID-19 (VIRA): Real-world logs from a vaccine information chatbot.
- Substance Use (SU): A curated KB from sources like the CDC and NIDA.
- Standard Benchmarks: MS MARCO and StackExchange.
- Adversarial Attacks: The authors used GPT-4o to generate "LLM-attacks" designed to mimic the style of the KB but with irrelevant or harmful intent.
- 4chan Attack: A dataset of "in-the-wild" toxic and off-topic queries used to stress-test the system.
Key Results: Performance and Efficiency
The lightweight detectors achieved OOD detection accuracy comparable to GPT-4o-based judges and established baselines like Mahalanobis distance or kNN.
- Latency: The PCA-based methods operate in microseconds (≈ 17-26 μs), whereas an LLM judge takes seconds (≈ 3.2 x 10^6 μs). This is a 100,000x speedup.
- Cost: The proposed method runs locally with near-zero marginal cost, while LLM judges incur significant API fees that scale with every user query.
- Robustness: The system remained effective even against "4chan-style" adversarial attacks that often bypass standard LLM guardrails.
- The Power of Low Dimensions: Effective OOD detection was achieved in fewer than 15 dimensions, proving that the full 768-dimensional embedding space is unnecessary for boundary detection.
Discussion and Limitations
Interestingly, the authors found that OOD queries don't always make a RAG system "lie" (factually incorrect), but they drastically reduce relevance. The system starts answering questions it wasn't designed for, which is a safety failure in specialized domains. Because the model uses PCA, the authors could "look under the hood," identifying specific principal components that corresponded to semantic themes like "Vaccine Eligibility."
However, some limitations remain. The evaluation was primarily English-focused. Additionally, the PCA is performed on a fixed KB; updates would require re-calculating the subspace. Finally, this method handles domain alignment (Axis A) but does not replace content moderation for harmful but "in-domain" requests (Axis B).
Summary updated