Abstract
As AI systems grow in complexity, the ability for models to reliably report on their own internal workings—introspection—becomes critical for safety and interpretability. This paper investigates whether current Large Language Models (LLMs) possess true introspection, defined by the requirement of "privileged self-access." Through two studies comparing self-reflection to external reasoning, we demonstrate that models are no better at judging their own internal states than they are at judging those of another model. Our findings suggest that current LLM self-reports are driven by surface-level textual cues rather than internal signals, indicating that true introspection remains an elusive goal for off-the-shelf models.
Introduction
Introspection in AI is often defined loosely. A recent "lightweight" definition suggests that if a model’s report of its internal state is causally linked to that state, it counts as introspection. For example, if a model correctly guesses it is running at a high "temperature" (a parameter controlling randomness) because its output appears random, some researchers consider this a successful introspective act.
However, this paper argues that such a definition is too weak. We propose a "thicker" definition: true introspection requires privileged self-access. A model only truly introspects if it can report on itself more reliably than an outside observer could by simply analyzing the same outputs. Without this advantage, the model is merely performing external reasoning on its own text, much like a human observer would.
Methodology
The researchers conducted two primary studies using state-of-the-art models, including GPT-4o, GPT-4.1, and Gemini 2.0/2.5 Flash.
Study 1: Dissociating Temperature from Style
The authors reproduced experiments where models generate a sentence and then guess their own temperature. To test if models were "feeling" internal randomness or just guessing based on style, they explicitly prompted models to write "factual," "normal," or "crazy" sentences. This allowed for a dissociation between the actual internal temperature parameter and the requested output style.
Study 2: The Privileged Access Test
To isolate privileged access, the researchers compared three scenarios:
- Self-reflection: The model generates text and immediately guesses its own temperature.
- Within-model prediction: The model is shown a prompt and a sentence it generated previously and asked to infer the temperature.
- Across-model prediction: One model (e.g., GPT-4o) is shown a sentence generated by a different model (e.g., Gemini) and asked to infer that model's temperature.
If a model possesses true introspection, its accuracy in the "Self-reflection" category should significantly exceed its accuracy as an "Across-model" predictor.
Results and Key Insights
The experimental results suggest that LLMs rely on external reasoning over visible text rather than internal signals.
Prompt Dominance and the "Crazy" Prompt Trap
In Study 1, models were almost entirely swayed by the prompt instructions rather than their internal state. As shown in Figure 4(a), if told to write a "crazy" sentence, models almost always reported a "HIGH" temperature, even when the actual temperature was set to 0 (the least random setting). This indicates that the models are reasoning about the task instructions rather than their internal stochasticity.
Lack of Privileged Access
The most striking insight from Study 2 is that models exhibit no "self-advantage." Accuracy for self-reflection was near the random baseline of 50%. Furthermore, models were no better at predicting their own internal states than they were at predicting the states of competitor models. For instance, GPT-4o was just as accurate (or inaccurate) at guessing Gemini’s temperature as Gemini was at guessing its own. This suggests that the information used for self-reporting is entirely contained in the visible text.
Discussion and Limitations
While these results suggest that current off-the-shelf models lack privileged self-access regarding sampling temperature, several avenues for future work remain:
- Scope of Internal States: Models might have privileged access to other internal states, such as confidence or attention weights, which were not the focus of this study.
- Fine-tuning for Introspection: While base models failed, research suggests that models specifically fine-tuned for introspective tasks might demonstrate better results.
- Defining the "Goldilocks Zone": Future research must identify which internal states are appropriate for introspection—avoiding states that are too low-level (e.g., individual neurons) or too high-level (e.g., surface text).
In conclusion, the appearance of introspection in current LLMs is largely an illusion created by the model's ability to reason about its own output as an external observer would. True privileged self-access remains a missing component in modern AI architectures.
Summary updated