Privileged Self-Access Matters for Introspection in AI

Siyuan Song, Harvey Lederman, Jennifer Hu, Kyle Mahowald

Abstract

As AI systems grow in complexity, the ability for models to reliably report on their own internal workings—introspection—becomes critical for safety and interpretability. This paper investigates whether current Large Language Models (LLMs) possess true introspection, defined by the requirement of "privileged self-access." Through two studies comparing self-reflection to external reasoning, we demonstrate that models are no better at judging their own internal states than they are at judging those of another model. Our findings suggest that current LLM self-reports are driven by surface-level textual cues rather than internal signals, indicating that true introspection remains an elusive goal for off-the-shelf models.

Introduction

Introspection in AI is often defined loosely. A recent "lightweight" definition suggests that if a model’s report of its internal state is causally linked to that state, it counts as introspection. For example, if a model correctly guesses it is running at a high "temperature" (a parameter controlling randomness) because its output appears random, some researchers consider this a successful introspective act.

However, this paper argues that such a definition is too weak. We propose a "thicker" definition: true introspection requires privileged self-access. A model only truly introspects if it can report on itself more reliably than an outside observer could by simply analyzing the same outputs. Without this advantage, the model is merely performing external reasoning on its own text, much like a human observer would.

Methodology

The researchers conducted two primary studies using state-of-the-art models, including GPT-4o, GPT-4.1, and Gemini 2.0/2.5 Flash.

Experimental setups for Study 1 (reflecting on generated content vs. internal state) and Study 2 (self-reflection vs. temperature prediction).

Study 1: Dissociating Temperature from Style

The authors reproduced experiments where models generate a sentence and then guess their own temperature. To test if models were "feeling" internal randomness or just guessing based on style, they explicitly prompted models to write "factual," "normal," or "crazy" sentences. This allowed for a dissociation between the actual internal temperature parameter and the requested output style.

Study 2: The Privileged Access Test

To isolate privileged access, the researchers compared three scenarios:

  • Self-reflection: The model generates text and immediately guesses its own temperature.
  • Within-model prediction: The model is shown a prompt and a sentence it generated previously and asked to infer the temperature.
  • Across-model prediction: One model (e.g., GPT-4o) is shown a sentence generated by a different model (e.g., Gemini) and asked to infer that model's temperature.

If a model possesses true introspection, its accuracy in the "Self-reflection" category should significantly exceed its accuracy as an "Across-model" predictor.

Results and Key Insights

The experimental results suggest that LLMs rely on external reasoning over visible text rather than internal signals.

Quantitative results from Study 1 (a) showing prompt dominance and Study 2 (b) showing a lack of privileged self-access across different models.

Prompt Dominance and the "Crazy" Prompt Trap

In Study 1, models were almost entirely swayed by the prompt instructions rather than their internal state. As shown in Figure 4(a), if told to write a "crazy" sentence, models almost always reported a "HIGH" temperature, even when the actual temperature was set to 0 (the least random setting). This indicates that the models are reasoning about the task instructions rather than their internal stochasticity.

Lack of Privileged Access

The most striking insight from Study 2 is that models exhibit no "self-advantage." Accuracy for self-reflection was near the random baseline of 50%. Furthermore, models were no better at predicting their own internal states than they were at predicting the states of competitor models. For instance, GPT-4o was just as accurate (or inaccurate) at guessing Gemini’s temperature as Gemini was at guessing its own. This suggests that the information used for self-reporting is entirely contained in the visible text.

Discussion and Limitations

While these results suggest that current off-the-shelf models lack privileged self-access regarding sampling temperature, several avenues for future work remain:

  • Scope of Internal States: Models might have privileged access to other internal states, such as confidence or attention weights, which were not the focus of this study.
  • Fine-tuning for Introspection: While base models failed, research suggests that models specifically fine-tuned for introspective tasks might demonstrate better results.
  • Defining the "Goldilocks Zone": Future research must identify which internal states are appropriate for introspection—avoiding states that are too low-level (e.g., individual neurons) or too high-level (e.g., surface text).

In conclusion, the appearance of introspection in current LLMs is largely an illusion created by the model's ability to reason about its own output as an external observer would. True privileged self-access remains a missing component in modern AI architectures.

Summary updated

BibTeX
@inproceedings{song2026privileged,
  author = {Siyuan Song and Harvey Lederman and Jennifer Hu and Kyle Mahowald},
  title = {{Privileged Self-Access Matters for Introspection in AI}},
  booktitle = {PhilML @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=ZcqCJHOWAA},
}
Figure