When Truthful Representations Flip Under Deceptive Instructions?

Xianxuan Long, Yao Fu, Runchao Li, Mu Sheng, Haotian Yu, Xiaotian Han, Pan Li

TL;DR

Large language models (LLMs) can be tricked into generating deceptive outputs by malicious instructions. This paper investigates how these instructions alter the LLM's internal workings, finding that deceptive prompts cause significant shifts in model representations, particularly in mid-to-late layers, and identifying specific "deception-associated features" that flip their activation patterns.

What Problem Does This Paper Address?

LLMs are increasingly capable of following complex instructions, which is a double-edged sword. While beneficial for many tasks, this ability also makes them susceptible to malicious instructions designed to elicit deceptive or harmful outputs. This poses a significant safety challenge. While we can observe the deceptive output, the underlying mechanisms within the LLM that lead to this deception are poorly understood. This paper aims to bridge this gap by investigating how deceptive instructions change the internal representations of LLMs, moving beyond just analyzing the final output.

Methods

The authors employ two primary techniques to analyze LLM internal representations:

  • Linear Probing: This standard technique involves training a simple linear classifier on the internal activations of an LLM at different layers. The goal is to see if the model's final "True" or "False" output can be predicted from these internal states. They use both standard Logistic Regression (LR) and a more nuanced method called Training of Truth and Polarity Direction (TTPD) to disentangle general output directions from other factors.
  • Sparse Autoencoders (SAEs): SAEs are used to decompose complex LLM representations into more interpretable, potentially "monosemantic" features. This allows for a finer-grained analysis of how specific features within the model's activations change under different instructions. They analyze the average SAE feature vectors and quantify shifts using metrics like L2 distance, cosine similarity, and feature overlap.

Datasets

The study uses two main categories of datasets for factual verification:

  • Curated Logical-Bench: This comprises six topic-specific datasets (cities, sp_en_trans, element_symb, animal_class, inventors, facts) with logical variants (negated, conjunction, disjunction). These datasets are designed to have syntactic homogeneity and minimal lexical noise.
  • Open-Domain Fact-Bench: This includes two datasets:
  • CommonClaim: Contains GPT-3-generated claims, adapted to balance true/false statements.
  • Counterfact: Consists of factual recall statements, with both true and false versions provided. These open-domain datasets are noisier and contain more topical diversity, testing the robustness of the findings.

Key Insights and Ideas

  • Consistent Output Predictability: Regardless of whether the instruction was truthful, neutral, or deceptive, the model's final "True" or "False" output was consistently predictable from its internal activations via linear probing. This suggests the core factual signal is preserved early on.
  • Deception is a Late-Stage Phenomenon: The authors found that deception isn't implemented by erasing factual content early in the model. Instead, it appears to be a late-stage change in token selection, likely happening in the mid-to-late layers.
  • Distinct "Honesty Subspace": SAE analysis revealed significant representational shifts under deceptive instructions, particularly in mid-to-late layers. These shifts are more pronounced than those seen with neutral prompts, suggesting a specific "reorientation" for deception.
  • "Deception-Associated Features" Flip Polarity: A small set of sparse features within the SAEs were identified as highly sensitive to deceptive instructions. These features systematically "flip" their activation patterns, becoming strongly active for one instruction type (truthful or deceptive) and suppressed for the other. These are termed "deception-associated features."
  • PCA Struggles with Complex Data: While PCA could visualize clear separations between "True" and "False" clusters on simple, curated datasets, it failed to do so on more complex, open-domain datasets due to feature superposition. SAEs proved more effective in disentangling these shifts.

Key Results

  • High Probing Accuracy: Linear probes achieved high accuracy in predicting the instructed output ("True"/"False") across all three prompt types (Truthful, Neutral, Deceptive) and layers, with peaks in mid-to-late layers (around layer 14 for LLaMA-3.1-8B and layer 21 for Gemma-2-9B).
  • Significant SAE Shifts: Deceptive prompts induced substantial shifts in SAE feature space, measured by L2 distance, cosine similarity, and feature overlap. These shifts were most pronounced in layers 10-16 for LLaMA and layers 16-32 for Gemma.
  • Robustness on Complex Datasets: The identified representational shifts persisted even on complex, uncurated datasets where global PCA failed to separate classes.
  • Few Key Features Drive Deception: Neuron-level analysis identified a small number of sparse SAE features that exhibited the largest activation changes between truthful and deceptive conditions. These features showed near-binary activation patterns, strongly supporting their role in deception.

Limitations and Future Work

  • English Declaratives Only: The study is limited to English declarative statements. Future work could extend to multilingual settings and more complex truth scales.
  • Frozen Model Weights: The experiments were conducted on frozen model weights. Future research could explore causal interventions (e.g., activation patching) or adversarial prompt recombinations.
  • Binary Factual Labels: The evaluation data was labeled for binary factuality. Extending this to graded truthfulness would be valuable.
  • Deception Detection and Mitigation: The identified "deception-associated features" offer a promising basis for developing future deception detectors and model editing techniques to mitigate instructed dishonesty.

Summary updated

Abstract

Large language models (LLMs) tend to follow maliciously crafted instructions to generate deceptive responses, posing safety challenges. How deceptive instructions alter the internal representations of LLM compared to truthful ones remains poorly understood beyond output analysis. To bridge this gap, we investigate when and how these representations ``flip'', such as from truthful to deceptive, under deceptive versus truthful/neutral instructions. Analyzing the internal representations of Llama-3.1-8B-Instruct and Gemma-2-9B-Instruct on a factual verification task, we find the model's instructed True/False output is predictable via linear probes across all conditions based on the internal representation. Further, we use Sparse Autoencoders (SAEs) to show that the Deceptive instructions induce significant representational shifts compared to Truthful/Neutral representations (which are similar), concentrated in early-to-mid layers and detectable even on complex datasets. We also identify specific SAE features highly sensitive to deceptive instruction and use targeted visualizations to confirm distinct truthful/deceptive representational subspaces. % Our analysis pinpoints layer-wise and feature-level correlates of instructed dishonesty, offering insights for LLM detection and control. Our findings expose feature- and layer-level signatures of deception, offering new insights for detecting and mitigating instructed dishonesty in LLMs.
BibTeX
@inproceedings{long2025when,
  author = {Xianxuan Long and Yao Fu and Runchao Li and Mu Sheng and Haotian Yu and Xiaotian Han and Pan Li},
  title = {{When Truthful Representations Flip Under Deceptive Instructions?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure