Where Should a Persona Live? Prompt versus Weights in Vision-Language Reasoning

Product teams increasingly want vision-language models (systems that read an image and a question and answer in text) to carry a personality: a patient, conscientious tutor who walks a student through a geometry diagram, or a warm assistant reading a chart aloud. The usual way to do this is to write the personality into the system prompt. In text-only models, this has a documented cost. Persona prompts degrade Theory-of-Mind reasoning, which is reasoning about what other people know or believe, and they bend factual judgments toward the persona's worldview. Separately, vision-language models are already weak and unstable at multi-step visual math, and small, meaning-preserving changes to the prompt flip their answers. Two things are unknown: whether a persona prompt makes this instability worse, and whether putting the personality into the weights instead of the prompt avoids the cost.

This note proposes a controlled comparison of two ways to give a vision-language model a Big Five personality. The first uses graded prompt descriptors. The second adds a scaled "personality vector" to the weights, meaning the weight difference between a trait-fine-tuned model and its base. We compare the two at matched measured trait strength on visual math and chart reasoning under systematic prompt perturbations. The central hypothesis is that persona prompts impose a reasoning tax that operates mainly through prompt sensitivity. On items where the model is already unsure, the extra persona instructions tip the answer. A weight-space persona of equal strength should leave both accuracy and answer stability close to the neutral model's.

Core Claim and What Is New

We make two claims. First, at equal trait strength as scored on the Big Five Inventory, a prompted persona lowers accuracy and raises answer volatility on visual reasoning benchmarks more than a weight-space persona does. Second, the prompt-induced losses concentrate on items where the neutral model's confidence in its answer is low. That pattern would link the persona tax to the mechanism that ProSA identifies for prompt sensitivity: small prompt shifts flip predictions when decoding confidence is weak.

The closest work is Personality Vector. It builds trait vectors by subtracting base weights from models fine-tuned on Big5-Chat and merges them into LLMs and vision-language models. It reports a correlation above 0.9 between the vector's scaling coefficient and BFI score. Its only multimodal evaluation, however, is subjective image preference on PsychoFlicker. It never asks what the vector does to structured reasoning or to prompt robustness, and it never compares against prompted personas at matched strength. Those three questions are what this proposal adds.

What Motivates and Constrains the Design

Several findings motivate the proposal. PHAnToM shows that Dark Triad persona prompts cut Llama 2's Answerability score on Theory-of-Mind tasks by 33.54%. Persona-Assigned LLMs Exhibit Motivated Reasoning shows accuracy collapsing when data contradict a persona's identity, with neither chain-of-thought nor "be accurate" instructions fixing it. Both studies are text-only and use prompting only. On the multimodal side, MATH-Vision attributes 42.2% of GPT-4V failures to reasoning rather than perception. Plot Twist shows chart accuracy swinging with titles and grid lines. Misaligned Roles, Misplaced Images shows that role-tag swaps and moving the image token shift internal representations of Qwen2-VL-7B and LLaVA-1.5.

Two results constrain what we can expect. The Thinking Spectrum finds that interpolating text models produces abrupt behavioral jumps at 60–70% mixing strength rather than smooth change. We should therefore expect the weight-space persona to stay harmless up to some coefficient and then break, and we sweep finely instead of assuming linearity. Personality Vector also reports interference when several trait vectors are merged at once. For that reason we study single traits first and treat multi-trait merging as secondary.

On the method side, we reuse two techniques. Linear interpolation of the form $W = W_{\text{base}} + \alpha\,(W_{\text{trait}} - W_{\text{base}})$, where $\alpha \ge 0$ scales the trait vector, follows WiSE-FT and PAINT. The graded-intensity prompt family follows Caron et al., who found persona expression scales linearly with the intensity of descriptors, and the psychometric shaping framework, which uses linguistic qualifiers to set levels.

Research Program

We use Qwen2-VL-7B and LLaVA-1.5, both of which appear in the perturbation literature above. For each model, we fine-tune its language backbone on each of the ten high/low trait splits of Big5-Chat. We then form trait vectors and add them to the vision-language model's language weights at $\alpha \in \{0, 0.1, \dots, 1.5\}$, following Personality Vector's transfer procedure. For the prompt condition, we build a five-level intensity ladder per trait using Caron-style qualifiers ("slightly," "very," "extremely"), plus the Dark Triad personas from PHAnToM.

Trait strength is measured the same way for both conditions. We administer the BFI and the Machine Personality Inventory from Evaluating and Inducing Personality in Pre-trained Language Models. We also run an LLM judge on the model's actual reasoning explanations, following the judging protocol of Can LLM Agents Maintain a Persona in Discourse?. Using the judge matters because a persona that shows up on a questionnaire but vanishes from the explanation would be a hollow win.

Every configuration is then evaluated on MATH-Vision, MathVista, and the Chart101 elementary questions. Each test item is presented under a fixed set of meaning-preserving perturbations: instruction paraphrases, reordered answer options, role-tag and image-position changes as in Misaligned Roles, and, for charts, removed or misleading titles and added grid lines. For each item we record three things: accuracy, ProSA's PromptSensiScore, and the ambiguity rate from Rethinking Hallucinations, which is the fraction of items whose answer changes across variants. We also record the neutral model's margin between its top two answer tokens as the confidence measure.

The main analysis plots accuracy and ambiguity against measured trait strength for both conditions and compares the two curves. The mechanism analysis bins items by neutral-model confidence and tests whether the persona-induced accuracy drop is concentrated in the low-confidence bins.

Baselines

The naive reference point is the neutral model with no persona, evaluated under the same perturbations. It fixes the accuracy and ambiguity floor that any persona method must be compared against. For orientation, MathVista reports that open models such as LLaVA scored 20–26% overall, against a 26.3% frequent-guess baseline and GPT-4V at 49.9%. MATH-Vision reports InternVL-Chat-V1-2-Plus at 16.97% and random chance near 5.86%. Because some models barely beat guessing, we report every effect as a paired difference from the neutral model on identical items.

The strongest existing way to deploy a persona is prompted personality with graded descriptors, in the style of Caron et al. and the psychometric shaping framework. It is strongest because it is what practitioners actually use and it needs no training. I found no reported result on Lacuna for persona-prompted vision-language models on MATH-Vision, MathVista, or Chart101. The nearest reported number is PHAnToM's −33.54% Answerability drop in text-only Theory-of-Mind. For a fair comparison, the prompt condition gets the same number of tuning trials as the α sweep, with each prompt level treated as one trial, and it is scored with the same trait instruments. We also add the anchored variant that ProSA found reduces sensitivity most, which includes one persona-consistent worked exemplar.

The strongest weight-space competitor is Personality Vector's recipe applied as published, with task-arithmetic merging and its reported coefficients. Our contribution here is evaluating it on reasoning, not changing it. We also include TIES and DARE merging from that paper, to check that any conclusion does not depend on one merging rule.

Two ablations isolate what drives any weight-space advantage. The first is a random direction with the same norm as the trait vector. It tests whether the effect is simply "any small weight perturbation." The second is a "pooled dialogue" vector fine-tuned on all ten Big5-Chat splits mixed together. It separates the trait direction from the side effects of chat fine-tuning itself, which could harm reasoning on its own.

Beating the prompt baseline means the following. At matched BFI score and matched judge-rated persona expression in explanations, the weight-space persona must show a smaller paired accuracy drop and lower ambiguity than the prompted persona. This must hold on at least two of the three benchmarks, with item-level bootstrap intervals that exclude zero, and the ablation vectors must not reproduce the result.

Why not just…?

Why not just solve neutrally, then rewrite the answer in the persona's voice? This is the most serious threat. A two-pass pipeline keeps reasoning accuracy at the neutral level by construction, so on final-answer accuracy it will match or beat our method. We include it as an upper reference, not something we expect to beat. What it costs is roughly twice the generation, and a persona that is applied after the fact rather than present during reasoning. That separation breaks down in multi-turn tutoring, where the persona must react to the student's intermediate replies. Our claim is therefore narrow. A single-pass weight-space persona should approach the pipeline's accuracy at half the inference cost while matching its judge-rated persona expression. If it falls well short of the pipeline, the practical case for weight-space personas on reasoning tasks fails, and we will report that.

Why not just add "reason carefully regardless of your persona" to the prompt? Motivated-reasoning work found that explicit accuracy instructions lowered veracity discernment by nearly 3% in persona-assigned models, and that chain-of-thought did not help. MATH-Vision separately reports that chain-of-thought is inconsistent for multimodal math. Those findings are text-only or persona-free, so we test the instruction directly as a prompt-condition variant rather than dismissing it.

Why not just fine-tune the vision-language model with persona data? Full fine-tuning on Big5-Chat is already the $\alpha = 1$ point of our sweep, so this alternative is included by design. What the sweep adds is the rest of the curve. WiSE-FT reports that interpolated weights beat end-point fine-tuning under distribution shift, with gains of 4 to 6 points on average and up to 23 on ImageNet-A. We expect an analogous interior optimum, but the Thinking Spectrum's phase jumps mean that optimum could be narrow.

Why not just detect unstable answers at run time and abstain? Sensitivity-based abstention like SENSEL, which improved selective-prediction AUC by 4.1 points over MaxProb on GPT-J-6B, flags fragile answers but does not remove the persona tax. It also multiplies inference cost. We report it as a complementary layer applied on top of both conditions.

Decisive Checks

The idea is supported only if three things hold. The weight-space persona must reach BFI and judge scores comparable to the strongest prompt level. At those matched scores, it must show a smaller paired accuracy drop and lower ambiguity on MATH-Vision, MathVista, and perturbed Chart101. And the prompt-induced losses must concentrate in the low-confidence item bins, while the random and pooled-dialogue vectors do not reproduce the benefit.

Three outcomes would falsify the hypothesis. If both conditions pay equal taxes, the cost comes from the personality itself rather than from where it is stored. If the tax is uniform across confidence bins, the prompt-sensitivity mechanism is wrong. If the weight-space persona only matches prompting after passing the Thinking Spectrum-style phase jump, where reasoning collapses, then weights offer no safe operating region.

A secondary check on Chart101 asks whether any persona, such as high conscientiousness, reduces the model's reliance on misleading titles. That would be a case where persona helps grounding rather than hurting it.