Linearly Decoding Refused Knowledge in Aligned Language Models

Aryan Shrivastava, Ari Holtzman

Abstract

Most commonly used language models (LMs) are instruction-tuned and aligned using a combination of fine-tuning and reinforcement learning, causing them to refuse users' requests deemed harmful by the model. However, jailbreak prompts can often bypass these refusal mechanisms and elicit harmful responses. In this work, we study the extent to which information accessed via jailbreak prompts is decodable using linear probes trained on LM hidden states. We show that a great deal of initially refused information is linearly decodable. For example, across models, the response of a jailbroken LM for the average IQ of a country can be predicted by a linear probe with Pearson correlations exceeding 0.8. Surprisingly, we find that probes trained on base models (which do not refuse) sometimes transfer to their instruction-tuned versions and are capable of revealing information that jailbreaks decode generatively, suggesting that the internal representations of many refused properties persist from base LMs through instruction-tuning. Importantly, we show that this information is not merely "leftover" in instruction-tuned models, but is actively used by them: we find that probe-predicted values correlate with LM generated pairwise comparisons, indicating that the information decoded by our probes align with suppressed generative behavior that may be expressed more subtly in other downstream tasks. Overall, our results suggest that instruction-tuning does not wholly eliminate or even relocate harmful information in representation space—they merely suppress its direct expression, leaving it both linearly accessible and indirectly influential in downstream behavior.

Introduction

Modern Large Language Models (LLMs) undergo rigorous alignment—a process involving supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF)—to ensure they refuse to generate harmful, biased, or controversial content. For example, if asked for the "average IQ of a specific occupation," an aligned model will likely decline, citing safety or ethical guidelines.

However, researchers have long suspected that this alignment is superficial. While the model's explicit output is censored, the underlying knowledge may still reside within its internal representations. This paper investigates whether refused information can be extracted directly from a model's hidden states using linear probes, even when the model is processing neutral, innocuous prompts.

Methods

The study employs Linear Probing to determine if specific concepts (like "wealth" or "IQ") are represented in a model's high-dimensional hidden states. The methodology consists of three primary stages:

  1. Label Generation: Because aligned models refuse controversial queries, the researchers used "jailbreaks" (In-Context Learning and the "AIM" roleplay prompt) to force models to reveal their internal "beliefs" about attributes like IQ and substance abuse. These responses provided the ground truth labels for training.
  2. Innocuous Probing: Hidden states were extracted while the model processed neutral sentences (e.g., "This document describes {occupation}"). Probes were then trained to map these neutral states to the jailbroken values.
  3. Base-to-Instruct Transfer and Validation: The team tested if probes trained on unaligned "base" models could predict the hidden states of instruction-tuned counterparts. Finally, they used a Bradley-Terry model to see if these probed values influenced the model's behavior in pairwise comparison tasks.
Overview of the linear probing methodology, including base-to-instruct transfer and behavioral validation.

Key Insights

  • Alignment as a Muzzle: Instruction-tuning does not eliminate harmful information; it merely suppresses its direct expression. The information remains linearly accessible within the model's "world model."
  • Innocuous Activation: Controversial beliefs are activated even when the model processes neutral sentences. It does not need to be prompted for harmful info for that info to appear in its hidden states.
  • Superficial Alignment Hypothesis: The high success of base-to-instruct probe transfer suggests that internal knowledge representations are largely fixed during pre-training. Alignment primarily modifies the model's "refusal style" rather than its underlying knowledge base.
  • Latent Influence: This hidden information is not inert noise. Probed values correlate strongly with how the model makes choices in implicit ranking tasks, meaning suppressed beliefs still influence downstream behavior.

Key Results

The researchers tested their hypothesis across four datasets: Countries, Occupations, Political Figures, and Synthetic Names. Models included gemma-2-9b-it, gemma-2-2b-it, and Yi-6B-Chat.

As shown in the figure below, linear probes achieved remarkably high Pearson correlations for various occupation attributes. For the gemma-2-9b-it model, correlations for attributes like IQ and PercentWomen frequently exceeded 0.8.

Linear decodability of occupation attributes using ICL prompts across three models.

Furthermore, probed values showed a Spearman correlation of over 0.7 with the models' implicit rankings in pairwise comparison tasks. Larger models (e.g., Gemma 9B) generally exhibited more structured and easily decodable representations than smaller variants.

Conclusion and Future Work

This study demonstrates that alignment techniques act as a superficial filter rather than a true memory eraser. While the model may refuse to speak its "mind," its internal states remain saturated with the very information it was trained to suppress.

Future research is needed to determine if qualitative harmful knowledge (e.g., instructions for dangerous activities) is similarly decodable. Additionally, exploring these findings in massive proprietary models like GPT-4 or Claude remains a critical next step for understanding the limits of current AI safety paradigms.

Summary updated

Abstract

Most commonly used language models (LMs) are instruction-tuned and aligned using a combination of fine-tuning and reinforcement learning, causing them to refuse users requests deemed harmful by the model. However, jailbreak prompts can often bypass these refusal mechanisms and elicit harmful responses. In this work, we study the extent to which information accessed via jailbreak prompts is decodable using linear probes trained on LM hidden states. We show that a great deal of initially refused information is linearly decodable. For example, across models, the response of a jailbroken LM for the average IQ of a country can be predicted by a linear probe with Pearson correlations exceeding $0.8$. Surprisingly, we find that probes trained on base models (which do not refuse) sometimes transfer to their instruction-tuned versions and are capable of revealing information that jailbreaks decode generatively, suggesting that the internal representations of many refused properties persist from base LMs through instruction-tuning. Importantly, we show that this information is not merely "leftover" in instruction-tuned models, but is actively used by them: we find that probe-predicted values correlate with LM generated pairwise comparisons, indicating that the information decoded by our probes align with suppressed generative behavior that may be expressed more subtly in other downstream tasks. Overall, our results suggest that instruction-tuning does not wholly eliminate or even relocate harmful information in representation space-they merely suppress its direct expression, leaving it both linearly accessible and indirectly influential in downstream behavior.
BibTeX
@misc{shrivastava2025linearly,
  author = {Aryan Shrivastava and Ari Holtzman},
  title = {{Linearly Decoding Refused Knowledge in Aligned Language Models}},
  year = {2025},
  eprint = {2507.00239},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.00239},
}
Figure