When a large language model is asked, with no training examples, to rate a person's personality from their writing, the answer for each trait often changes when the instruction is reworded, the options are reordered, or the examples are shuffled. The standard response is to average over rewordings or to throw the unstable cases away. We propose using that instability as information instead. Rephrase the prompt several times and record how much each trait's rating wobbles for each person. Treat the wobble as that trait's measurement noise. Then combine the five noisy ratings with the known pattern of how human traits co-vary, using a standard Bayesian noisy-measurement correction. A trait the model reads confidently, such as Extraversion, then pulls an unstable one, such as Neuroticism or Agreeableness, toward values consistent with it. The pull is strong exactly where the model was unreliable and near zero where it was stable.
This matters because two known failures meet here. Zero-shot personality inference, where a frozen model is prompted without examples, predicts each trait in isolation and degrades badly at finer granularity. Prompt sensitivity is worst exactly where the model's internal confidence is flat. Psychometrics offers a free constraint that neither literature uses: the Big Five traits (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) are correlated in human populations. The method needs only black-box API access and no fine-tuning. It also produces a diagnostic of its own: whether the model's errors are correlated across traits, a "halo" effect in which one impression colors all five ratings.
Core Claim
A per-person, per-trait noise estimate obtained from prompt perturbations, combined with a population trait-covariance prior, should improve continuous and three-class Big Five estimates from frozen models. It should do so beyond what the same number of API calls buys through plain averaging. The gain should concentrate on the person–trait pairs the model is least stable on.
Concretely, let $y \in \mathbb{R}^5$ be a person's true trait scores, with population prior mean $\mu$ and covariance $\Sigma$. Let $x \in \mathbb{R}^5$ be the model's mean rating across $K$ reworded prompts, and $D$ the $5\times5$ covariance of those $K$ ratings divided by $K$. The corrected estimate is the Gaussian posterior mean
$$\hat{y} = \mu + \Sigma(\Sigma + \alpha D + \Lambda)^{-1}(x - b - \mu),$$
where $b$ is the model's systematic bias per trait, $\Lambda$ is a diagonal floor for noise that rewording does not reveal, and $\alpha$ scales perturbation variance into true error variance. Without labels, $b=0$ and $\alpha$, $\Lambda$ are set by a fixed rule. With a small labeled calibration split of about 50 people, $b$, $\alpha$ and $\Lambda$ are fitted by least squares.
$D$ is a full matrix. If rewording moves several traits together, which is a halo signature, the correction discounts that shared movement rather than double-counting it.
Relation to Prior Work
The gap is motivated by three findings. Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation prompts GPT-3 once per trait on Facebook posts. It finds that Macro F1 falls from 0.454 (binary) to 0.230 (three-class), and it notes a bias toward high Openness and low Neuroticism. Its traits never see each other, and it explicitly defers prompt composition effects. Affective-NLI reframes conversational personality recognition as entailment and acknowledges that it treats the traits as independent. On Text-based Personality Computing counts only 5 of 60 ACL studies that model traits jointly, all of them supervised.
The joint-modeling literature supplies the idea but not a usable method. Contrastive Graph Transformer Network for Personality Detection decodes traits autoregressively. Computational personality recognition in social media uses stacking and multi-objective forests. Both require training on labeled users. We keep their premise that traits co-constrain each other but apply it at inference time to a frozen model.
The sensitivity literature supplies the noise signal. ProSA ties sensitivity to flat decoding confidence and finds that one demonstration reduces sensitivity the most. On the Relation between Sensitivity and Accuracy in In-Context Learning finds sensitivity correlates with error (Pearson $r$ up to −0.40) and uses it only to abstain (SENSEL). We keep the SENSEL signal but use it to correct predictions rather than to discard them, and we use it per output dimension of a structured target, which that work does not consider.
Two papers constrain the design. Stable and Explainable Personality Trait Evaluation … with Internal Activations shows that questionnaire-based trait scores swing by 10–20 points under rewording and cuts the standard deviation from 7.20 to 0.73 with activation vectors. That method needs model weights and measures the model's own persona, not a human author's, so it is not a direct competitor. Personality as a Probe for LLM Evaluation finds Openness and Conscientiousness entangled inside models. This warns that model errors may be correlated across traits, which is why $D$ is a full matrix.
A further constraint comes from the context-gap finding in Computational personality recognition in social media: relations between cues and traits can invert across platforms. It is therefore an open empirical question whether $\Sigma$ transfers across corpora, and we test it directly.
Research Program
The work has three parts.
The first is elicitation. For each person and trait, we obtain a continuous rating as the probability-weighted mean over rating tokens 1–5 where logits are available, and a parsed generated number otherwise. The two are reported separately because The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models reports that logit and text readouts disagree more than 50% of the time for Llama-2. The prompts follow the ITEMDESC style (trait descriptions built from questionnaire items) from the GPT-3 study. We use $K \in \{4, 8, 16\}$ variants spanning paraphrase, option order, and ProSA-style prompt styles.
The second is the prior. We estimate $\Sigma$ from labels on a disjoint split of the same corpus, and separately from a different corpus, to measure transfer.
The third is diagnosis. We compare the covariance of model errors across traits with the human trait covariance to test for halo, and we measure whether error magnitude tracks the diagonal of $D$. The whole method would fail if that link did not hold.
Evaluation uses datasets from the file. The main two are Essays (2,468 authors with Big Five labels) and the Big Five subset of PANDORA (Reddit). We also run on the Chinese counselling dialogues via the released code, where Predicting the Big Five … Chinese Counselling Dialogues reports zero-shot Qwen1.5-110B at 0.425 average Pearson correlation. If the Jose et al. Facebook set of 142 users is obtainable, we replicate the GPT-3 comparison. The models are one open model with logits, such as Llama 3-70B-Instruct (the most stable model in ProSA), and one API model.
The metrics are per-trait Pearson correlation, three-class Macro F1 using the GPT-3 study's binning, a variance-across-rewordings score for final predictions, and the rate of implausible profiles (joint Mahalanobis distance above the 99th percentile of the human distribution).
Baselines
The naive reference is a single ITEMDESC prompt per trait, the setup of Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation. For three-class Macro F1 the reported reference is 0.230 on that paper's Facebook set. On Essays and PANDORA we found no reported zero-shot LLM result on Lacuna, so we measure it ourselves.
The strongest compute-matched competitor is $K$-prompt averaging: the same $K$ rewordings, with the ratings averaged per trait. This is exactly our estimate with the prior removed. It isolates the key ingredient and uses the identical API budget. We also run its calibrated version, fitting the per-trait bias $b$ and a scale on the same 50-person split, so that any gain cannot be attributed to calibration alone.
The second competitor is few-shot anchoring. Following ProSA, we give one or three labeled demonstrations drawn from the calibration split, with no correction.
The third is a joint prompt that asks for all five ratings at once, optionally in a fixed chain where each trait sees the earlier answers. This is the obvious prompt-level version of inter-trait modeling. On counselling data the zero-shot reference is 0.425 Pearson correlation (Qwen1.5-110B, from the counselling paper).
For label-rich context we report the supervised ridge regression on n-grams and topics (WT-LEX, which beat GPT-3 in the GPT-3 study) and the fine-tuned 0.582 Pearson correlation from the counselling paper. These are upper references, not targets.
Two ablations accompany the method. One uses a diagonal $\Sigma$, which is plain shrinkage toward the mean. The other uses a trait-shuffled $\Sigma$ that keeps the correlation magnitudes but permutes which traits they connect.
Beating the baselines means a statistically reliable improvement, by paired bootstrap over people, in mean Pearson correlation and three-class Macro F1 over calibrated $K$-averaging at equal $K$ on at least two of the three corpora. The shuffled-$\Sigma$ ablation must lose that gain.
Why not just…?
Why not just average more rewordings? Averaging reduces variance but not bias. The GPT-3 study's systematic high-Openness, low-Neuroticism tilt survives any number of rewordings. Averaging also cannot use the fact that Extraversion is informative about another trait. The averaging baseline has an identical call budget, and the covariance-free ablation shows directly how much of any gain is only extra calls. If the ablation matches the full method, the covariance idea is refuted.
Why not just ask for all five traits in one prompt, or chain them? This is the real threat, because a capable model may already reason jointly. Its weakness is that the model's implicit trait relations need not match human ones. The entanglement found in Personality as a Probe for LLM Evaluation and the assistant-like Agreeableness inflation reported by TRAIT suggest the model's joint ratings would carry a halo rather than human covariance. A chain also passes the first trait's error to every later trait. The halo diagnostic tests this directly. If joint prompting produces errors that are nearly independent across traits and matches our accuracy, we will report that the explicit prior adds nothing for that model.
Why not just use few-shot demonstrations? ProSA's evidence that one demonstration reduces sensitivity is real, and anchoring is complementary rather than competing. We therefore also apply the correction on top of few-shot prompts. The claim is that residual instability after anchoring is still informative noise. If few-shot outputs become so stable that $D$ carries no signal about error, that part of the claim fails.
Why not just abstain on unstable cases, as SENSEL does? Abstention is fine for classification but unacceptable when every person needs a full five-trait profile. We report the coverage–accuracy curve of SENSEL-style abstention against our corrected predictions at full coverage.
Decisive Checks
Four checks decide the proposal.
The first tests the mechanism. Stratify person–trait pairs by their perturbation variance. The improvement over averaging should grow monotonically with variance and be near zero in the most stable quintile. A flat profile would mean the gain comes from generic shrinkage, not from using sensitivity.
The second tests the size of the gain against a ceiling. Because Big Five correlations in humans are modest, the maximum gain is limited by how much of an unstable trait's variance the other four traits explain. We compute that ceiling from $\Sigma$ and report the fraction of it achieved. A gain far above the ceiling would indicate leakage.
The third tests transfer. Using a $\Sigma$ estimated on one corpus and applied to another should keep most of the gain. If it does not, the method needs in-domain labels, and we will say so.
The fourth checks agreement between readouts. We test whether the correction reduces the disagreement between logit-based and text-based readouts reported in The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models. This is a secondary outcome, and we make no prediction about its direction.
References
- Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation
- Affective-NLI: Towards Accurate and Interpretable Personality Recognition in Conversation
- On Text-based Personality Computing: Challenges and Future Directions
- Contrastive Graph Transformer Network for Personality Detection
- Computational personality recognition in social media
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- Stable and Explainable Personality Trait Evaluation in Large Language Models with Internal Activations
- Personality as a Probe for LLM Evaluation: Method Trade-offs and Downstream Effects
- The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models
- Predicting the Big Five Personality Traits in Chinese Counselling Dialogues Using Large Language Models
- Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics