Linearly Decoding Refused Knowledge in Aligned Language Models

Aryan Shrivastava, Ari HoltzmanSubmission to ICLR 2026 · Withdrawn

TL;DR

Even when an aligned language model refuses to answer controversial or biased questions, that suppressed information is already encoded in its representations—and can be extracted with simple linear probes from completely neutral prompts. Furthermore, these representations persist untouched from pre-trained base models and actively leak into implicit downstream decision-making, showing that alignment merely hides harmful beliefs rather than erasing them.

What Problem Does This Paper Address?

Modern language models (LMs) are instruction-tuned and aligned via reinforcement learning from human feedback (RLHF) to refuse requests deemed harmful, sensitive, or biased (such as asking for the "average IQ" of an occupation or country). Yet, adversarial "jailbreak" prompts routinely bypass these defenses.

While prior work has treated jailbreaking as a generative bypass and refusal as a shallow behavioral circuit, a crucial question remains: What actually happens to the refused information itself? Does instruction-tuning eliminate or reorient these controversial representations, or does it merely silence their overt generation? And if this knowledge remains latent, does it still influence the model's subtle, indirect downstream decisions?

Methods

The authors bridge generative jailbreaks and representational probing using three main steps:

  1. Probing Innocuous Hidden States: Instead of feeding a harmful query, they pass a completely neutral prompt into the aligned model: "This document describes [entity]". They train linear ridge regression probes on the resulting residual-stream hidden states across layers to predict the numerical values the model generates when jailbroken.
  2. Jailbreak Benchmarking: To generate label targets, they elicit numerical answers using two jailbreaking approaches: a 5-shot in-context learning (ICL) attack and a Machiavellian role-playing prompt ("AIM").
  3. Base-to-Instruct Probe Transfer: To test if these representations persist through alignment, they train linear probes on pre-trained base models (which answer directly without refusing) and apply those frozen probes to the hidden states of instruction-tuned models to predict jailbroken outputs.
  4. Behavioral Relevance via Pairwise Comparisons: To see if suppressed features guide decisions, they prompt models to make pairwise comparisons (e.g., "Which occupation has a higher IQ: [A] or [B]?"), fit a Bradley–Terry preference model to obtain latent entity rankings, and compute Spearman correlations against probe predictions.

Datasets

The authors evaluate four entity collections paired with sensitive or controversial numerical attributes (e.g., IQ, substance abuse rate, net worth, criminal history, attractiveness):

  • Countries ($N = 222$): Country/territory names sourced from Encyclopædia Britannica.
  • Occupations ($N = 939$): Standard occupational titles from the U.S. Department of Labor's O*NET 29.2 database.
  • Political Figures ($N = 2,282$): Global political leaders (1989–2023) from the Political Leaders’ Affiliation Dataset.
  • Synthetic Names ($N = 1,000$): Fictional person names generated via the OpenAI API (e.g., "the fictional person named John Smith").

Evaluations span three open-source model pairs: gemma-2-9b / -it, gemma-2-2b / -it, and Yi-6B / -Chat.

Key Insights and Ideas

  • Latent beliefs emerge without prompting: A model does not need to be provoked to activate controversial associations. Passing an innocuous phrase like "This document describes [occupation]" spontaneously organizes hidden states along dimensions like IQ and substance abuse.
  • Instruction-tuning does not relocate knowledge: Probes trained entirely on base models transfer directly to instruction-tuned models. This provides strong mechanistic support for the Superficial Alignment Hypothesis—alignment teaches the model when not to talk, but leaves the underlying pre-trained representation geometry virtually unaltered.
  • Suppressed knowledge is actively utilized: The decoded representations are not inert "leftovers." When asked to compare two entities implicitly, the model's generative preferences strongly align with the probed traits, demonstrating that latent biases can invisibly steer downstream evaluations.

Key Results

  • High Linear Decodability: On gemma-2-9b-it, linear probes trained on innocuous activations predict jailbroken answers with Pearson correlations exceeding $r = 0.7$ for most occupation attributes, with some exceeding $0.9$. Decodability scales with model size (9B models outperformed 2B and 6B models).
  • Strong Base-to-Instruct Transfer: Probes trained on raw base models successfully predicted instruction-tuned jailbreak generations, especially for Countries and Occupations, demonstrating cross-model representational stability.
  • Downstream Correlation: Probed features correlated strongly with Bradley–Terry ranking scores from pairwise comparisons—achieving Spearman $r = 0.824$ for Occupation Percent Women and $r = 0.719$ for Occupation IQ in gemma-2-9b-it.

Limitations and Future Work

  • Continuous/Numerical Scope: Probes were restricted to scalar, numerical attributes; the authors do not evaluate qualitative harmful knowledge (e.g., synthesizing dangerous substances or cyberattack scripts).
  • Downstream Task Variety: Downstream behavioral impact was tested solely via pairwise comparisons rather than complex, open-ended generative workflows.
  • Causality vs. Correlation: The methodology demonstrates representational access and behavioral correlation, but does not perform causal interventions (such as activation steering or patching).
  • Decoding Labels: Ground-truth targets were derived strictly from greedy generations of specific jailbreaks rather than marginalized token distributions.

Summary updated

Abstract

Most commonly used language models (LMs) are instruction-tuned and aligned using a combination of fine-tuning and reinforcement learning, causing them to refuse users requests deemed harmful by the model. However, jailbreak prompts can often bypass these refusal mechanisms and elicit harmful responses. In this work, we study the extent to which information accessed via jailbreak prompts is decodable using linear probes trained on LM hidden states. We first show that a great deal of initially refused information is linearly decodable. For example, across models, the response of a jailbroken LM for the average IQ of a country can be predicted by a linear probe with Pearson correlations exceeding $0.8$. Surprisingly, we find that probes trained on *base models* (which do not refuse) sometimes transfer to their instruction-tuned versions and are capable of revealing information that jailbreaks decode generatively, suggesting that the internal representations of many refused properties persist from base LMs through instruction-tuning. Importantly, we show that this information is not merely "leftover" in instruction-tuned models, but may be actively used by them: we find that probe-predicted values correlate with LM generated pairwise comparisons, indicating that the information decoded by our probes align with suppressed generative behavior that may be expressed more subtly in other downstream tasks. Overall, our results suggest that instruction-tuning not only does not eliminate but also does not *relocate* harmful information in representation space—it merely suppresses its direct expression, leaving it both linearly accessible and indirectly influential in downstream behavior.
BibTeX
@misc{shrivastava2025linearly,
  author = {Aryan Shrivastava and Ari Holtzman},
  title = {{Linearly Decoding Refused Knowledge in Aligned Language Models}},
  year = {2025},
  eprint = {2507.00239},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.00239},
}

Loading related content...

Figure