Represented but misread: Post-training changes how language models read out what they represent

Jochen LaubrockarXiv 2026 · arXiv

TL;DR

Language models often "know" more than they say. This paper demonstrates that post-training (like supervised fine-tuning) primarily changes how a model "reads out" its internal representations into an answer, rather than changing the representations themselves. Surprisingly, this process can actually make models worse at certain tasks, such as identifying AI-generated text, by teaching them to ignore or misinterpret valid internal cues.

What Problem Does This Paper Address?

There is a growing debate in AI research known as the superficial alignment hypothesis: the idea that large language models (LLMs) acquire most of their knowledge during massive pre-training, and that subsequent post-training (SFT, DPO, etc.) merely teaches them the style and format in which to respond.

However, we often see a "knowledge-behavior gap." For example, an LLM might write perfectly like an AI but fail to identify AI-written text when asked. This paper investigates whether post-training changes what a model represents (its internal "knowledge") or how it reads out that representation into a final answer. The authors use authorship detection (Human vs. AI) and truth-telling (especially in negated statements) as test cases to pinpoint where in the network this "misreading" occurs.

Methods

The researchers employed a "two-lens" framework based on Brunswik’s lens model from psychology to separate encoding (how the world is reflected in internal states) from readout (how those states are used to form an answer).

Figure 1. Brunswik’s lens model, applied to a language model as judge and extended by an internal lens. A state of the world (Brunswik’s distal criterion; the ground truth, here whether a text was written by an AI, whether a statement is true, or whether a review is genuine) is reflected in observable cues of the text, some of which are valid and some of which are not (dashed). In the first lens (encoding), the model integrates these cues into its internal representation; whether the ground truth can be decoded from this representation is measured with linear probes, and whether two checkpoints represent it in the same way with probe transfer. In the second lens (readout), the representation is read out into an answer to the question by weighting internal features. Only one of these features was measured directly: the class direction, i.e., the direction that separates the two classes in the representation; its relation to the answer was examined with steering at the answer position (Appendix D). The other features are schematic. Achievement, i.e., the agreement of the answer with the state of the world, requires both lenses. Post-training changes mainly the second lens (orange), and in several cases also the first (Sections 3.4 and 3.6); model stitching and activation patching locate the change in depth and in the prompt. Splitting the judge side of the lens model has precedents in personality judgment (detection vs. utilization of cues; Funder, 1995) and in neuroscience (information encoded in a neural population vs. read out for behavior; Panzeri et al., 2017). Note: for the authorship corpora, probes were fitted both on the text positions and on the last prompt position; for the truth and review sets, only on the last prompt position.

To analyze this, they used several sophisticated interpretability interventions:

  • Linear Probes: Training simple classifiers on the model's internal activations to see if the information (e.g., "is this AI-written?") is present, even if the model doesn't say it.
  • Model Stitching: Creating "hybrid" models by taking the bottom layers of a base model and the top layers of a fine-tuned model (or vice versa) to see which layers control the change in behavior.
  • Activation Patching: Swapping the internal states of specific tokens (like the question or the text itself) between base and fine-tuned versions to locate the "locus" of the change.
  • Signal Detection Theory (SDT): Using the $d'$ metric to measure a model's true sensitivity to a signal, independent of its bias toward answering "human" or "AI."
Figure 2. How the interventions work. In panels A, C, and D, the horizontal axis shows the positions of the prompt in reading order (the instruction, the text to be judged, the question, and the position at which the answer is produced), and the vertical axis shows the layers of the network. (A) Information flow: through attention, later positions read information from earlier ones. The answer is produced at the last position. Information from the text reaches it through the positions after the text: the question (dark tiles) and the last position itself, which also reads the text directly at every layer (the schematic shows only the route via the question). (B) Steering, sketched in a two-dimensional version of the high- dimensional representation space: human texts (open circles) and AI texts (triangles) differ along an AI–human direction (arrow between the class means). Steering moves the state of one text along this direction (diamonds) and measures how the model’s answer changes; the dashed line marks where the answer switches. The two need not be orthogonal: the angle between the AI–human direction and the model’s decision boundary determines how strongly a shift along the direction changes the answer. If the model read the direction correctly, a shift toward the AI texts would move the answer toward “KI”. (C) Model stitching combines the lower layers of one checkpoint (here the base model, blue) with the upper layers of another (here the SFT model, orange). (D) Activation patching replaces the internal states of one model (here SFT) at selected positions and one layer with those of the other model (here the base model’s states at the question positions).

Datasets

The study utilized three primary corpora of German texts:

  1. Chat messages: Student-written posts vs. AI-generated counterparts (naive, style-matched, and human-optimized).
  2. Newspaper segments: Local journalism vs. AI-generated articles.
  3. Textbook summaries: Human-written summaries vs. ChatGPT-generated ones.

They also used public datasets for validation:

  • Truth Judgments: 748 matched pairs of true/false statements, including negated versions (e.g., "The city of Bokaro is not in China").
  • Deceptive Reviews: The Deceptive Opinion Spam Corpus (hotel reviews).

Key Insights and Ideas

  • Information is Available but Ignored: In almost every model, linear probes could identify AI-written text with high accuracy, yet the models' actual answers were often no better than chance—or even inversely correlated with the truth.
  • Post-training Can Impair Sensitivity: For authorship detection, post-training actually lowered the models' sensitivity ($d'$). The models didn't just become more biased; they became worse at using the cues they already had.
  • The "Locus" of Change: Model stitching revealed that the weights responsible for these changes often reside in the early layers, but activation patching showed that the "repair" or "impairment" is carried by the tokens following the text (the question and answer cue), not the text tokens themselves.
  • The Power of Small Signals: A training study showed that adding just 2,000 labeled examples during fine-tuning could completely flip the readout direction. If the training data contains a "misleading" signal for a specific genre, the model will faithfully learn to misread its own internal states.

Key Results

  • Authorship Sensitivity Drop: Across six model families, post-training decreased the sensitivity of answers by an average of $\Delta d' = -0.34$.
  • Negation Repair: Base models were notoriously bad at judging the truth of negated statements (often ordering them inversely). Post-training "repaired" this readout, bringing the answers in line with the internal representations which were already at "ceiling" accuracy (as shown by probes).
  • Base Rate Overweighting: Base models were "conservative" and ignored stated base rates (e.g., "80% of these texts are AI"). Post-trained models swung to the opposite extreme, often overweighting the base rate compared to an ideal observer.
Figure 3. Comparison of sensitivity ($d'$) between internal probes (top) and actual model answers (bottom). Note how probes are much more sensitive than answers.

Limitations and Future Work

  • Model Scale: The study focused on models in the 4B to 12B parameter range. It remains to be seen if massive frontier models (100B+) exhibit the same "misreading" patterns or if they develop better "introspection."
  • Language Specificity: The authorship corpora were primarily German. While the truth/negation results on English data were consistent, broader cross-linguistic studies are needed.
  • Causal Origins: While the authors showed that training data can set the readout direction, they couldn't pinpoint exactly which parts of standard, massive SFT datasets (like those used for Llama or Gemma) cause the specific decline in authorship detection. Identifying these "poisonous" or misleading samples is a key next step for safer alignment.

Summary updated

Abstract

Does post-training change what a language model represents, or how this representation is read out into an answer? We compared linear probes with the models' own answers, analyzed with signal detection theory, in six open-weight base models and their post-trained versions. For judgments of whether German texts were written by a human or an AI, probes and word n-gram classifiers separated the classes well, but the answers did not and often ordered the texts inversely. Post-training lowered the sensitivity of the answers on average ($Δd' = -0.34$ across model families; -0.29 across three prompts). It also changed the use of a stated base rate: Base models shifted their log-odds by less than the stated change in prior odds, most post-trained models by more. In two models with stage-wise checkpoints, the change occurred during supervised fine-tuning, model stitching implicated the early layers, and from mid-depth on internal states at positions after the text, not those of the text, transferred the change between checkpoints. Our corpora cannot be shared, so we add a case reproducible from public data, the truth of negated statements. Here, post-training acted in the opposite direction: Probes decoded truth at or near ceiling in every model, the answers of several base models did not use it, and post-training repaired them; by mid-depth the repair resided at the positions after the statement; below it, the representation of the statement contributed, strongly in one of four models. In a training study, adding 2,000 correctly labeled authorship judgments of news texts to otherwise identical fine-tuning data improved the answers on our newspaper and textbook corpora and inverted them on the chats, where they followed a news-trained probe of the base model. Post-training thus changed how a judgment is read out of what a model represents, and a small training signal can set its direction.
BibTeX
@misc{laubrock2026represented,
  author = {Jochen Laubrock},
  title = {{Represented but misread: Post-training changes how language models read out what they represent}},
  year = {2026},
  eprint = {2611.26584},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26584},
}
Figure