SAFE: A Sparse Autoencoder-Based Framework for Robust Query Enrichment and Hallucination Mitigation in LLMs

Samir Abdaljalil, Filippo Pallucchini, Andrea Seveso, Hasan Kurban, Fabio Mercorio, Erchin SerpedinEMNLP 2025

TL;DR

This paper introduces SAFE, a novel framework that uses Sparse Autoencoders (SAEs) to detect and mitigate hallucinations in Large Language Models (LLMs). By enriching queries with interpretable features extracted by SAEs, SAFE significantly improves response accuracy and reduces factual inaccuracies, achieving up to a 29.45% accuracy improvement.

What Problem Does This Paper Address?

Large Language Models (LLMs) are powerful tools for generating human-like text, but they suffer from a critical flaw: hallucinations. These are instances where LLMs fabricate information, blurring the line between truth and fiction. This unreliability poses a significant challenge for critical applications. Hallucinations can be broadly categorized into factual (information beyond training data) and relevance (factually correct but contextually irrelevant) types. While existing mitigation strategies often focus on data or input augmentation (like RAG), they tend to overlook the internal workings of LLMs, which are believed to be a root cause of hallucinations. Specifically, polysemanticity, where single neurons activate across multiple unrelated concepts, obscures the model's decision-making process.

Methods

SAFE tackles hallucinations through a two-stage process:

  1. Hallucination Detection: This stage is modular and can integrate with any existing hallucination detection method that provides a confidence or hallucination score. The system uses a configurable threshold to flag responses with a high risk of hallucination. The authors demonstrate this by integrating SAFE with three state-of-the-art detectors: SINdex, HaloCheck, and SelfCheckGPT.
  2. Query Enrichment with SAEs: For flagged responses, SAFE employs Sparse Autoencoders (SAEs) to enrich the original query. SAEs are unsupervised methods that decompose neural network activations into interpretable, monosemantic features. The core idea is that instead of injecting new knowledge, SAFE steers the LLM to better leverage its existing knowledge by selecting the most relevant features. This involves:
  3. Extracting sparse, contextually important features from both the question and response using SAEs.
  4. Identifying response-specific features that are not present in the question context.
  5. Measuring the semantic similarity of these response-specific features to the question using sentence-BERT embeddings.
  6. Discarding outlier features that might be misleading.
  7. Recomposing the query with these enriched, relevant features.

This process is repeated iteratively until the hallucination score falls below a predefined threshold.

Datasets

The evaluation of SAFE was conducted on four diverse cross-domain datasets:

  • TruthfulQA: Designed to assess LLM performance on questions that challenge common misconceptions.
  • BioASQ: A biomedical QA dataset containing yes/no and open-ended questions.
  • WikiDoc: A medical QA dataset from a platform for sharing medical knowledge.
  • HaluEval: A QA hallucination detection benchmark dataset comprising general queries.

The authors report using a randomly sampled subset of 400 questions from each dataset.

Key Insights and Ideas

  • Leveraging Interpretability for Mitigation: The paper's central insight is that by using interpretable features extracted by SAEs, we can guide LLMs to better utilize their existing knowledge, thereby reducing hallucinations. This moves beyond simply filtering data or augmenting inputs.
  • SAEs as a Tool for Knowledge Steering: SAEs are shown to be effective in decomposing complex neural activations into understandable components, which can then be used to "steer" the LLM's attention towards more relevant information.
  • Training-Free Approach: SAFE is designed as a plug-and-play solution that integrates into existing LLM pipelines without requiring additional model fine-tuning, making it highly practical.
  • Synergy of Detection and Enrichment: The combination of a robust hallucination detection mechanism with SAE-guided query enrichment is crucial for SAFE's effectiveness.

Key Results

  • SAFE consistently improves query generation accuracy and mitigates hallucinations across all tested datasets.
  • Accuracy improvements of up to 29.45% were observed, particularly with the Llama3-8b model on the TruthfulQA dataset when using SINdex for detection.
  • SAFE demonstrated effectiveness across different LLMs (Gemma2-9b and Llama3-8b) and various hallucination detection methods (SINdex, HaloCheck, SelfCheckGPT).
  • When compared to simpler enrichment methods like adding a generic prompt or using chain-of-thought prompting, SAFE consistently outperformed them.
  • The framework showed that enhancing smaller models with SAFE can yield performance comparable to or exceeding that of larger models in many cases.

Limitations and Future Work

  • LLM Architecture Dependency: SAFE's reliance on SAEs means its applicability is currently limited to LLMs that expose internal representations amenable to SAE analysis. Extending this to models without such characteristics would require modifications or alternative interpretability techniques.
  • Query Quality Influence: The effectiveness of SAFE is inherently influenced by the quality of the input queries. Low-quality queries may still lead to suboptimal performance, a common challenge in LLM systems.
  • Language Restriction: The current implementation is restricted to English-language inputs.
  • Future Directions: The authors suggest exploring multilingual and multimodal extensions for SAFE.

Summary updated

BibTeX
@inproceedings{abdaljalil2025safe,
  author = {Samir Abdaljalil and Filippo Pallucchini and Andrea Seveso and Hasan Kurban and Fabio Mercorio and Erchin Serpedin},
  title = {{SAFE: A Sparse Autoencoder-Based Framework for Robust Query Enrichment and Hallucination Mitigation in LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure