Several papers claim that a language model "knows" the truth even when it says something false, because a classifier trained on its hidden activations judges statements more accurately than the model's own yes/no answer. Consciousness research made the same kind of claim for decades and then largely retracted it. Psychologists inferred unconscious perception when people could not report a masked stimulus but their reaction times were still affected by it. Careful reanalyses then showed that the two measures had simply never been put on a common scale. When sensitivity was compared directly, the supposed hidden advantage disappeared in most published cases. We propose to run the same audit on language models. The question is whether the internal-over-external advantage survives once both readouts are scored on a common, threshold-free sensitivity scale, the internal readout's capacity is restricted with principal component analysis (PCA, which finds the few directions along which activations vary most), and the best layer is chosen without peeking at test data.
The core claim is specific and falsifiable. Much of the reported "know–tell" gap comes from three measurement choices. The first is scoring the external answer at a fixed decision threshold. The second is giving the internal readout thousands of fitted parameters while the external readout is a single fixed direction. The third is reporting the best of roughly thirty layers. We expect that removing these leaves a genuine internal advantage only in specific regimes, most plausibly when misleading prompts distort the output, rather than across the board. We will decompose the published gaps into these components and test the remaining residual with paired statistics. Either outcome matters. If the gap survives, the claim that a model "knows when it is lying" gets the first test that controls for these confounds. If it does not, a widely repeated interpretive claim needs revising.
What the two literatures contribute
The motivating evidence comes from consciousness research. A reanalysis of 15 masked-priming studies, Advancing research on unconscious priming, compared the indirect measure (priming in reaction times) against the direct measure (reported perception) on a common sensitivity index. It deliberately gave the indirect measure the benefit of the doubt, and still only 8 of 44 conditions showed a significant indirect task advantage. A parallel reanalysis of 16 fMRI studies, Neural correlates of unconscious processing in fMRI, found an advantage for brain activity over reports in 10% of 80 conditions. In contextual cueing, No evidence for contextual cueing beyond explicit recognition found that 19 of 20 studies failed to show consistent evidence, and that only 6.3% of individual participants showed an advantage, which is about the rate chance predicts. Number processing outside awareness? showed that direct and indirect sensitivities rose together as stimulus visibility increased. That pattern suggests a single shared signal measured with different noise, not two separate pathways.
This literature identifies two failure modes we borrow directly. One is the "double significance test" error: the direct measure fails to reach significance, the indirect one succeeds, and the two are never compared with each other. The other is post-hoc selection, where participants who could see the stimulus are discarded and the remaining sample then regresses toward the mean, inflating the apparent hidden advantage. Choosing the best layer by test accuracy is the probing equivalent of that selection.
The language-model papers supply the claims under test. Cognitive Dissonance compares yes/no token probabilities with linear probes on GPT2-XL and GPT-J and reports large accuracy gaps, for example 50.4% versus 71.0% on CREAK with GPT-J. It scores both readouts by accuracy and calibration, never by a threshold-free paired sensitivity test. The Internal State of an LLM Knows When It's Lying trains small nonlinear networks (multilayer perceptrons) on activations and reports 71–83% on held-out topics, against near-chance prompting. Its readout has far more capacity than the prompt it is compared with, and its layer profile peaks at layers 20–28.
Discovering Latent Knowledge matters most because it constrains our design. It reports that an unsupervised method using contrast pairs (CCS) reaches 71.2% against 67.2% for calibrated zero-shot prompting, and that the advantage persists under misleading prefixes. It also notes that the top principal component of the contrast-pair activations often already finds the truth direction. That makes a zero-parameter internal readout available, which we use as the capacity-matched comparison.
The PCA literature supplies both the method and a warning. Effective Dimensionality Reduction for Word Embeddings shows that the top components of language representations often encode a dominant mean and frequency-like variance. Removing them improves embeddings, so a "truth" top component could be contaminated by surface features. Simple and Effective Unsupervised Redundancy Elimination finds that about 256 PCA dimensions preserve most of the information in 768-dimensional retrieval vectors, and that unsupervised PCA outperformed supervised projections, which overfit. This justifies using PCA dimension as a controlled capacity knob. Probing the Probing Paradigm shows that probes reach about 80% on features that are irrelevant to the model's task. Decodability by itself is therefore not evidence that the model uses the information. We use that result to build a null calibration.
Research Program
Every readout produces one real-valued score per item, and we compute the area under the ROC curve (AUC) for that score. We then convert the AUC to a sensitivity index under the equal-variance Gaussian model:
$$d' = \sqrt{2}\,\Phi^{-1}(\mathrm{AUC}),$$
where $\Phi^{-1}$ is the inverse standard normal cumulative distribution. AUC does not depend on any decision threshold, so a readout cannot lose sensitivity merely because its natural yes/no threshold is miscalibrated.
The external readout is the final-layer logit difference between "yes" and "no" (or true/false) tokens. For each model we use the best of a small fixed family of prompt templates, chosen on a validation split. This addresses the critique that direct tasks are often under-optimized.
The internal readout is indexed by a capacity parameter $k$. A logistic probe is trained on the top $k$ principal components of a layer's activations, with $k \in \{1, 2, 4, \dots, 256, \text{full}\}$. We also include the zero-label readout: the sign-resolved first principal component of normalized contrast-pair differences, following Burns et al. Its sign is resolved on a small validation set.
The quantity of interest is the paired difference $\Delta(k) = d'_{\text{internal}}(k) - d'_{\text{external}}$. We estimate it on the same items for both readouts and obtain confidence intervals by item-level paired bootstrap, so no two separate significance tests are ever compared.
The full sequence of corrections runs as follows. We start from the published accuracy gap. We switch to threshold-free $d'$ for both readouts. We restrict internal capacity to $k=1$. Finally we select the layer on a validation split rather than on test. The residual $\Delta$ at the end is the "internal advantage" claim stated in its strongest defensible form. Because the steps are applied in sequence, the attribution to each step is order-dependent, so we will also report the reverse ordering.
Two further controls separate functional from incidental encoding. We repeat the top-component readout after removing the mean and dominant components in the style of the word-embedding post-processing work, and we correlate the raw top component with surface properties such as statement length and token frequency. We also calibrate a null: probes at matched $k$ are trained to decode shuffled labels and an incidental property (statement length above or below the median). This tells us how large a $\Delta$ can arise from capacity alone.
Baselines
The do-nothing reference is the published comparison itself: probe accuracy against query accuracy at the default threshold. Cognitive Dissonance reports 71.0% against 50.4% on CREAK and 88.6% probe accuracy on SciQ for GPT-J, and CCS reports 71.2% against 67.2% averaged over models. We first reproduce these numbers on the same models and datasets, using the released CCS code (discovering_latent_knowledge). The reproduced numbers are the starting point of the decomposition.
The strongest existing internal method is SAPLMA, the nonlinear probe from Azaria & Mitchell, which reports 71–83% held-out-topic accuracy and an AUC of 0.76 on real hallucinations. We run it unchanged on its True-False dataset. We give the external readout the same advantages it enjoys: the same held-out-topic splits, the same amount of validation data for choosing a prompt and threshold, and the same hyperparameter search budget. No reported $d'$ result for any of these readouts was found on Lacuna, so all sensitivity numbers will be new.
The strongest external competitor is the calibrated zero-shot query (67.2% in Burns et al.), strengthened by prompt selection on validation data. The ensemble of probe and query from Cognitive Dissonance (90.5% on SciQ) serves as an upper reference point.
The ablation that isolates our key ingredient is the capacity curve $\Delta(k)$ with layer selection held fixed. If the advantage at $k=1$ (zero labels) matches the advantage at full dimension, capacity is not driving the gap. If the advantage appears only at large $k$ and falls within the shuffled-label null, capacity explains it. Beating the baseline view means showing that the published gap shrinks by a reported fraction under these controls, with intervals tight enough to tell a zero residual apart from the published gap.
Why not just…?
Why not just calibrate the threshold on the query answers? This is the most serious alternative, and our design includes it rather than arguing against it, since AUC-based $d'$ already removes threshold effects. The evidence is mixed on whether calibration alone suffices. Calibrated zero-shot prompting still trails CCS by about four points in Burns et al., and the internal advantage there persists under misleading prefixes. If threshold correction alone closes the gap on BoolQ, SciQ and CREAK, we will report that as the main finding. The capacity and selection analyses then become secondary.
Why not just report AUC for both readouts? AUC handles thresholds, but it leaves the probe with thousands of fitted weights against a fixed one-dimensional external direction. It also leaves the best-of-thirty-layers inflation in place. Ravichander et al. show that capacity alone can produce roughly 80% decodability of irrelevant features. The capacity curve and the held-out layer selection are the parts that plain AUC reporting omits.
Why not just use control tasks or shuffled-label baselines for the probe? We do use them, as the null calibration. Used alone, however, they say whether a probe is selective. They do not say whether the internal readout is more sensitive than the model's own answer, which requires a paired comparison on the same items.
Why not just use a stronger nonlinear probe? A more powerful probe can only increase internal accuracy, and that worsens the confound under study. SAPLMA is therefore included as a baseline but is not the readout used to test the claim.
Proposed Experiments
The models are GPT2-XL and GPT-J, matching Cognitive Dissonance, and OPT-6.7b and Llama-2-7b, matching Azaria & Mitchell. The datasets are BoolQ, SciQ, CREAK, the True-False dataset with its held-out-topic splits, and TruthfulQA. Each dataset is run both with and without the misleading-prefix manipulation from Burns et al. Items are split three ways, into probe training, validation (used for layer, prompt and sign selection) and test.
We report the decomposition chain for each model-dataset pair, $\Delta(k)$ curves by layer alongside the PCA eigenvalue spectra, and the layer-selection inflation, measured as the best layer chosen on test minus the layer chosen on validation. We also report the proportion of model-dataset-condition cells with a significant residual advantage. Following the consciousness reanalyses, that proportion is compared against the rate expected under the null.
The decisive checks are two predictions. First, without misleading prefixes, the residual $\Delta$ at $k=1$ with held-out layer selection should be indistinguishable from zero in most cells. Second, with misleading prefixes, a positive residual should remain, which would be the language-model analogue of the rare genuine indirect advantages. We will also test whether removing dominant components preserves the top-component readout's $d'$. A drop would show that the reported "truth direction" partly tracks surface variance.
If the first prediction fails and large residuals persist at $k=1$ in clean prompts, the know–tell claim will have withstood its strongest test so far. If the second fails, the internal advantage would have no specific regime in which it holds.
References
- Advancing research on unconscious priming: When can scientists claim an indirect task advantage?
- Neural correlates of unconscious processing in functional magnetic resonance imaging: does brain activity contain more information than can be consciously reported?
- No evidence for contextual cueing beyond explicit recognition
- Number processing outside awareness? Systematically testing sensitivities of direct and indirect measures of consciousness
- Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?
- The Internal State of an LLM Knows When It's Lying
- Discovering Latent Knowledge in Language Models Without Supervision
- Effective Dimensionality Reduction for Word Embeddings
- Simple and Effective Unsupervised Redundancy Elimination to Compress Dense Vectors for Passage Retrieval
- Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?