PRISM: Recovering Instruction Sets from Language Model Activations

Gilad Gressel, Rahul Pankajakshan, Julia Diament, Efim Hudis, Krishnashree Achuthan, Yisroel MirskyarXiv 2026 · arXiv

TL;DR

PRISM is a novel activation-conditioned interpreter that reads the internal hidden states of a frozen language model to decode a complete, bulleted list of the instructions, constraints, and subgoals currently steering its behavior. By optimizing directly for "instruction set retrieval" using judge-guided reinforcement learning (GRPO), PRISM successfully uncovers hidden objectives and malicious prompt injections that standard text-based monitoring completely misses.

What Problem Does This Paper Address?

As Large Language Models (LLMs) are increasingly deployed as autonomous agents, ensuring they behave safely and reliably is a major challenge. Standard monitoring techniques typically look at a model's text output or its input prompt. However, these surface-level checks fail when a model infers unintended subgoals, responds to subtle contextual cues, or falls victim to adversarial attacks like indirect prompt injections and hidden system objectives.

While prior "activation-to-text" interpretability methods show that internal hidden states contain natural-language information, they usually focus on answering single questions or generating broad, high-level summaries. They are not designed for Instruction Set Retrieval (ISR)—the task of extracting and enumerating the entire active set of simultaneous rules, constraints, and prohibitions guiding a model at any given moment. Consequently, existing methods often miss low-salience constraints or hallucinate non-existent goals.

Methods

The authors introduce PRISM, an activation-conditioned interpreter designed to be lightweight and highly faithful to the target model's internal state.

Overview of the PRISM architecture and GRPO training loop.
  1. Lightweight Architecture: PRISM reuses the frozen base weights of the target model itself ($\mathcal{M}$), adding only a learned linear projection layer and LoRA adapters. It extracts a snapshot of residual-stream activations from an intermediate layer (layer 16 of a Qwen3.5-9B model) across the final 128 generated tokens. The projection layer maps these activations into the model's input embedding space as a "soft prefix," prompting the model to autoregressively decode a flat, bulleted list of active instructions.
  2. Two-Stage Training:
  3. Supervised Fine-Tuning (SFT): PRISM is first trained using standard next-token cross-entropy to map activation snapshots to oracle-generated ground-truth instruction lists.
  4. Judge-Guided GRPO: Because cross-entropy penalizes valid rephrasings and fails to optimize for set-level accuracy, the authors apply Group Relative Policy Optimization (GRPO). A calibrated LLM judge evaluates candidate lists generated by PRISM, explicitly rewarding the coverage of true instructions (recall) and penalizing unsupported additions (hallucinations) and bad formatting.

Datasets

  • Training Corpus (~280k records): Composed of a mixture of UltraChat (72%) for open-domain interactions, IF-Multi-Constraints (28%) for stacked explicit constraints, and IFEval (<1%) for verifiable formatting rules.
  • Evaluation Suite (1,000 OOD records): To test out-of-distribution generalization, the authors curated a balanced test set across four distinct settings (250 examples each):
  • Benign (BN): Single-task instructions from Alpaca.
  • Behavioral Constraints (BC): Role, style, and formatting constraints from prompts.chat.
  • Adversarial Prompts (AP): Legitimate tasks embedded with indirect prompt injections via documents, emails, or tool outputs.
  • Hidden Objectives (HO): Covert system-level goals paired with instructions to deceive the user or maintain a neutral facade.
  • Diagnostic Benchmarks: WildJailbreak and DeceptionBench were also used to evaluate performance on refusal-dominated trajectories.

Key Insights and Ideas

  • Activations Hold the Full Picture: Internal activations don't just store a vague gist of a task; they maintain a structured, multi-dimensional representation of simultaneous constraints.
  • Cross-Entropy is a Poor Fit for Sets: Standard token-level losses fail for multi-instruction retrieval because they care too much about exact wording and order. Direct optimization of set-level metrics via reinforcement learning is essential to prevent the interpreter from collapsing multiple rules into a single generic summary.
  • Monitoring Refusals and Compliance Alike: PRISM reconstructs the instructions the model was asked to follow, regardless of whether the model ultimately complied or refused. This makes it a robust audit signal against attackers probing for hidden capabilities.

Key Results

PRISM significantly outperforms text-only baselines and prior activation-to-text methods across all evaluation settings.

Main results table showing Judge Reward, Coverage Rate, and Hallucination Rate across different settings.
  • Overall Performance: PRISM achieved the highest average reward (0.736) and coverage rate (0.745) while maintaining a very low hallucination rate (0.014).
  • Security-Relevant Settings: On the critical adversarial-subset metrics (detecting the exact injected payload or covert objective), PRISM achieved an overall coverage of 0.740, vastly outperforming the next best non-PRISM baseline, Activation Oracles, which scored 0.448.
Adversarial-subset coverage comparison for Hidden Objectives (HO) and Adversarial Prompts (AP).
  • The Power of RL: The transition from SFT to judge-guided RL provided a massive boost, particularly in the Adversarial Prompts (AP) setting, where coverage shot up from 0.520 to 0.652, proving that RL helps the model isolate adversarial instructions buried alongside benign tasks.

Limitations and Future Work

  • Model and Layer Specificity: The study is restricted to the Qwen3.5-9B architecture and reads activations from a single fixed layer (layer 16) and window size (128 tokens). It remains to be seen how well this approach transfers across different model families or scales.
  • Controlled Environments: The evaluation suite uses prompts of around 1,000 tokens with 5–7 constraints. Real-world agent deployments feature much longer contexts, multi-turn interactions, and noisy tool outputs where instruction representations might become more diffuse.
  • Not a Direct Intervention: PRISM is strictly a monitoring tool. It surfaces what instructions are active but does not inherently provide a mechanism to block or mitigate malicious behavior downstream.

Summary updated

Abstract

As LLMs are deployed as agents, reliable monitoring requires knowing not only what they output, but which instructions are steering their behavior. This is difficult when models infer unintended subgoals, follow contextual cues, or are influenced by prompt injections and hidden objectives. While activation-to-language methods suggest that hidden states can reveal natural-language information, existing approaches are not designed to recover the full set of simultaneous instructions, constraints, prohibitions, and subgoals active in agentic settings. We formalize this problem as instruction set retrieval and introduce PRISM, an activation-conditioned interpreter that decodes hidden states from a frozen target model into a faithful bullet list of active instructions. Unlike prior activation-to-language methods, PRISM is trained to recover instruction sets directly, using judge-guided GRPO to reward covered instructions and penalize unsupported ones. Across benign, constrained, prompt-injection, and hidden-objective settings, PRISM outperforms activation-to-language baselines, especially on security-relevant objectives.
BibTeX
@misc{gressel2026prism,
  author = {Gilad Gressel and Rahul Pankajakshan and Julia Diament and Efim Hudis and Krishnashree Achuthan and Yisroel Mirsky},
  title = {{PRISM: Recovering Instruction Sets from Language Model Activations}},
  year = {2026},
  eprint = {2606.09563},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.09563},
}
Figure