Knowing When Not to Answer: Lightweight KB-Aligned OOD Detection for Safe RAG

Ilias Triantafyllopoulos, Renyi Qu, Salvatore Giorgi, Brenda Curtis, Lyle H. Ungar, João Sedoc

TL;DR

Retrieval-Augmented Generation (RAG) systems often "hallucinate" when asked questions outside their specific knowledge base (KB). This paper introduces a lightweight, PCA-based gatekeeper that detects out-of-domain (OOD) queries in microseconds, matching the accuracy of expensive LLM-based judges while being significantly cheaper and more interpretable.

What Problem Does This Paper Address?

RAG systems are designed to ground AI responses in a specific set of documents. However, they suffer from a "forced choice" problem: if a user asks a question completely unrelated to the knowledge base, the retriever will still surface the "least-irrelevant" documents it can find. This often leads the generator to produce fluent but entirely unjustified or unsafe responses—a major risk in high-stakes fields like medicine or public health.

While developers often use "guardrails" or additional LLM calls to judge query relevance, these methods are computationally expensive, slow (adding seconds of latency), and act as "black boxes." There is a clear need for an "always-on" gate that is fast, cheap, and explains why a query was rejected.

Methods

The authors propose a straightforward but effective geometric approach to OOD detection:

  1. Embedding & PCA: They take the embeddings of the Knowledge Base (KB) and apply Principal Component Analysis (PCA) to identify the "shape" of the in-domain data.
  2. Subspace Selection: Instead of just taking the top components, they compare two strategies:
  3. Explained Variance Retention (EVR): Keeping components with the most data spread (standard PCA).
  4. p-value Ranking: Using a t-test to identify which specific principal components best separate in-domain queries from OOD queries.
  5. Low-Dimensional Projection: Queries are projected into a very compact subspace (often $<15$ dimensions).
  6. Geometric Classifiers: They test simple rules like $\epsilon$-ball (is the query within a certain radius of KB samples?) and $\epsilon$-rect, alongside lightweight machine learning models like Gaussian Mixture Models (GMMs) and SVMs.

Datasets

The study evaluates the method across 16 domains, with a focus on high-stakes environments:

  • COVID-19 (VIRA): Real-world logs from a vaccine information chatbot.
  • Substance Use (SU): A curated KB from sources like the CDC and NIDA.
  • Standard Benchmarks: MS MARCO and StackExchange.
  • Adversarial Attacks: The authors used GPT-4o to generate "LLM-attacks" designed to mimic the style of the KB but with irrelevant or harmful intent.
  • 4chan Attack: A dataset of "in-the-wild" toxic and off-topic queries used to stress-test the system.

Key Insights and Ideas

  • Relevance vs. Correctness: Interestingly, the authors found that OOD queries don't always make a RAG system "lie" (factually incorrect), but they drastically reduce relevance. The system starts answering questions it wasn't designed for, which is a safety failure in specialized domains.
  • The Power of Low Dimensions: You don't need 768-dimensional embeddings to find the "boundary" of a domain. Most OOD detection happened effectively in fewer than 15 dimensions.
  • Separability > Variance: The p-value ranking often outperformed standard EVR. This suggests that the dimensions where the data varies the most aren't necessarily the ones that define the "edge" of the knowledge base.
  • Interpretability: Because the model uses PCA, the authors could "look under the hood." They identified that specific principal components corresponded to clear semantic themes, such as "Vaccine Eligibility" or "Systemic Distrust."

Key Results

  • Performance: Lightweight detectors (specifically GMMs) achieved OOD detection accuracy comparable to GPT-4o-based judges.
  • Latency: The PCA-based methods operate in microseconds ($\approx 17-26 \mu s$), whereas an LLM judge takes seconds ($\approx 3.2 \times 10^6 \mu s$). This is a $100,000\times$ speedup.
  • Cost: The proposed method runs locally with near-zero marginal cost, while LLM judges incur significant API fees that scale with every single user query.
  • Robustness: The system remained effective even against "4chan-style" adversarial attacks that often bypass standard LLM guardrails.

Limitations and Future Work

  • Language Constraints: The evaluation was primarily focused on English-language datasets. While the COVID-19 data included Spanish, the adversarial tests were English-only.
  • Static vs. Dynamic KBs: The PCA is performed on a fixed KB. If the knowledge base is updated frequently, the subspace might need to be re-calculated, though the authors note this is an "offline" cost.
  • Policy vs. Alignment: This method handles "Axis A" (is this in the KB?). It does not replace "Axis B" (is this query toxic or against policy?). A complete safety stack would still require a separate content moderation filter for harmful but "in-domain" requests.

Summary updated

Abstract

Retrieval-Augmented Generation (RAG) systems are increasingly deployed in high-stakes domains, where safety depends not only on how a system answers, but also on whether a query should be answered given a knowledge base (KB). Out-of-domain (OOD) queries can cause dense retrieval to surface weakly related context and lead the generator to produce fluent but unjustified responses. We study lightweight, KB-aligned OOD detection as an always-on gate for RAG systems. Our approach applies PCA to KB embeddings and scores queries in a compact subspace selected either by explained-variance retention (EVR) or by a separability-driven t-test ranking. We evaluate geometric semantic-search rules and lightweight classifiers across 16 domains, including high-stakes COVID-19 and Substance Use KBs, and stress-test robustness using both LLM-generated attacks and an in-the-wild 4chan attack. We find that low-dimensional detectors achieve competitive OOD performance while being faster, cheaper, and more interpretable than prompted LLM-based judges. Finally, human and LLM-based evaluations show that OOD queries primarily degrade the relevance of RAG outputs, showing the need for efficient external OOD detection to maintain safe, in-scope behavior.
BibTeX
@inproceedings{triantafyllopoulos2026knowing,
  author = {Ilias Triantafyllopoulos and Renyi Qu and Salvatore Giorgi and Brenda Curtis and Lyle H. Ungar and João Sedoc},
  title = {{Knowing When Not to Answer: Lightweight KB-Aligned OOD Detection for Safe RAG}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.740},
}
Figure