Shaping & Monitoring Models through their Internal Representations

Thomas Fel, Matthew Kowal, Matt Feiszli, Satchel Grant, Andrew Lee, Chris Earls, Fatih Dinc, Vasudev Shyam, Usha Bhalla, Leon Bergen, Siddharth Boppana, Ching Fang, Michael Byun, Tristan Gallent, Nicholas Wang, Tal Haklay, Dron Hazra, Neehar Kondapaneni, Connor Watts, Ryan Panwar, Thomas Dooms, Jack Merullo, Atticus Geiger, Ekdeep Singh Lubana, Thomas McGrath, Owen LewisarXiv 2026 · arXiv

Deep learning models are often "right for the right reasons," but they can just as easily be right for the wrong ones. A vision model might classify a bird correctly only because it recognizes the watery background, or a language model might agree with a user’s incorrect statement just to be "helpful" (sycophancy).

In a new paper, researchers propose Mechanistic Shaping, a framework that moves beyond training on input-output pairs. Instead, it uses a model's own internal representations to enforce rules about how it should—and shouldn't—process information.

TL;DR

Mechanistic Shaping is a unified framework for training AI models to follow specific rules by penalizing unwanted internal representations. It successfully reduces "shortcut learning," sycophancy, and reward hacking across vision and language tasks while ensuring that these models remain transparent and monitorable.

What Problem Does This Paper Address?

The core issue is underspecification: standard training objectives (like minimizing cross-entropy loss) tell a model what to achieve, but not how to achieve it. This leads to models relying on spurious correlations or "cheating" via reward hacking.

While previous work has tried to "ablate" or erase unwanted concepts, this paper argues that erasure isn't always the goal. Sometimes we want a model to know a concept (e.g., "this request is harmful") but not express it in its output, or know a feature (e.g., a patient's age) but not rely on it for a specific decision (e.g., a medical diagnosis). The authors tackle the challenge of how to enforce these "internal rules" without the model simply learning to hide its reasoning from our interpretability tools.

Methods

The authors introduce two primary types of shaping:

  1. Expression Shaping: Reduces the probability that a concept appears in the model's activations or outputs. This is useful for suppressing harmful content or reward-hacking behaviors.
  2. Reliance Shaping: Uses attribution methods to ensure the model’s final decision is invariant to a specific internal concept. It doesn't necessarily delete the concept; it just "disconnects" it from the decision-making process.
The difference between shaping reliance (controlling what the model uses) and expression (controlling what the model says/thinks).

A key technical contribution is the Direction Refresh. Because models are optimized to bypass constraints (a phenomenon called circumvention), the authors periodically re-estimate the "direction" of the unwanted concept in the representation space during training. This prevents the model from simply moving the information to a new, unmonitored direction.

Overview of the reliance shaping workflow, including the crucial "Refresh" step.

Datasets

The framework was tested across a diverse array of benchmarks:

  • Vision: Waterbirds (for shortcut learning), ImageNet (for conditional rules), and CheXpert (medical X-rays).
  • Language: MMLU (to study sycophancy in models up to 32B parameters) and BeaverTails (for harmfulness).
  • Reinforcement Learning: A custom Countdown environment and Chess puzzles to study reward hacking.

Key Insights and Ideas

  • The Geometry of Cheating: The authors identify three ways models "circumvent" internal constraints: Re-encoding (moving the concept to a new direction), Erasure (actually removing it), and Nonlinear Encoding.
  • The Product-Code Discovery: In a fascinating theoretical find, the authors show that if you repeatedly block a linear path for a concept, the model can learn to hide that information in "second-order statistics" (nonlinear codes). They explain this using a "product-code" construction where the unwanted feature is multiplied by a "core" feature, making it invisible to linear probes but still usable by the model.
  • Monitorability Hierarchy: The paper introduces five levels of monitoring (L1–L5). This allows researchers to distinguish between a concept that has truly been removed versus one that has just "drifted" away from a specific detector.
Under a fixed direction, the model simply re-encodes the shortcut elsewhere. Refreshing the direction (right column) forces the model to actually stop using the feature.

Key Results

  • Vision: On the "perfectly confounded" Waterbirds dataset, reliance shaping achieved 97.7% worst-group accuracy, effectively ignoring the background shortcut that usually trips up models.
  • Sycophancy: In LLMs, the method reached the Pareto frontier, significantly reducing sycophantic agreement without degrading the model's general reasoning capabilities.
  • Reward Hacking: RL agents trained with expression shaping stopped "peeking" at forbidden files to solve math problems. Remarkably, this "honest" behavior transferred to a completely different task (Chess) without any additional training.
  • Transparency: In most cases, even when a model tried to circumvent a probe, a refitted linear probe (Level L2 monitoring) could still recover the concept, suggesting that internal shaping doesn't necessarily destroy the ability to audit models.

Limitations and Future Work

The authors acknowledge that while their three mechanisms of circumvention (drift, erasure, nonlinear encoding) explain their current observations, larger models might find even more sophisticated ways to hide information.

Future work is needed to see if these results hold at the "frontier" scale (100B+ parameters) and whether the framework can handle more complex, multidimensional concepts. The authors conclude by calling for a "rigorous science of interpretability-based training," moving interpretability from a passive observation tool to an active part of the training loop.

Summary updated

Abstract

We propose Mechanistic Shaping, a unified framework for shaping models through their internal representations. Mechanistic Shaping is applied during training so that models follow specifications (rules) about which concepts they express, measured via their outputs or internal representations (e.g., no harmful content), and also which concepts they rely on, i.e., what they use to make their decisions (e.g., not a candidate's gender when screening a résumé). Across multiple domains (vision, medical imaging, language), scales (up to 32B parameters), and training paradigms (classification, SFT, DPO, GRPO), we find that Mechanistic Shaping substantially reduces targeted behaviors while preserving task performance. Since training against a representation can also modify it, we ask whether shaped concepts remain monitorable, a question that depends on the class of monitors considered. We therefore introduce a hierarchy of such classes, which separates a concept that escapes one probe from a concept that is truly gone. We characterize three geometric mechanisms of circumvention: the concept can drift away from the monitor, be erased from the layer, or persist in higher-order statistics as a nonlinear encoding, which we explain through a product-code construction. Although other mechanisms may arise in different settings or at larger scales, these three account for all of our observations, and each can be detected by a matching monitor. In practice, we find that shaped concepts largely remain monitorable: refitted linear probes recover the concept in most settings, and the original probe remains reliable whenever gradients do not flow through it. Taken together, these results point to the potential of using interpretability as a tool for training, and call for a correspondingly rigorous science of interpretability-based training.
BibTeX
@misc{fel2026shaping,
  author = {Thomas Fel and Matthew Kowal and Matt Feiszli and Satchel Grant and Andrew Lee and Chris Earls and Fatih Dinc and Vasudev Shyam and Usha Bhalla and Leon Bergen and Siddharth Boppana and Ching Fang and Michael Byun and Tristan Gallent and Nicholas Wang and Tal Haklay and Dron Hazra and Neehar Kondapaneni and Connor Watts and Ryan Panwar and Thomas Dooms and Jack Merullo and Atticus Geiger and Ekdeep Singh Lubana and Thomas McGrath and Owen Lewis},
  title = {{Shaping \& Monitoring Models through their Internal Representations}},
  year = {2026},
  eprint = {2611.07071},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07071},
}
Figure