Lexical Gaps as a Driver of Order Bias in Persona-Based LLM Recommenders

Large language model (LLM) recommenders increasingly read a compact text summary of a user, a "persona" such as "likes hard science fiction, dislikes romance," and then choose among candidate items listed in the prompt. Two failures are well documented in isolation. These models favor items by where they sit in the list rather than by what they are, which is called positional or order bias. Separately, the words used to describe a user's interests often differ from the words used in item descriptions, which is called vocabulary mismatch or the lexical gap. This note proposes that the two are linked. When the model cannot match a candidate's wording to anything in the persona, it becomes uncertain, and uncertain models fall back on position. If that is true, part of order bias in persona-driven recommendation is a symptom of the lexical gap. Rewriting the persona in the catalog's own vocabulary should then make the recommender less sensitive to order at the cost of a single forward pass, compared with averaging over many shuffled prompts.

The question matters because the obvious remedies are expensive or weak. Averaging over permutations multiplies inference cost by the number of orderings. Prompts that tell the model to "consider all options equally" do not work. Heuristic reordering can cancel itself out. A cause located in the input text could instead be fixed in the input text, offline, when the persona is built and cached.

Core Claim

In persona-conditioned listwise recommendation, we expect a candidate's sensitivity to order to be predicted by its low surface-word overlap with the persona, even after controlling for its semantic similarity to the persona as measured by an embedding model. We further expect that a persona rewritten into catalog vocabulary, with meaning held fixed, reduces order sensitivity and recovers part of the accuracy that permutation averaging recovers, at roughly one-kth of the cost of averaging over k orderings.

The claim builds on two strands of motivating evidence. First, the selection-bias literature shows that models resort to positional guessing when they lack the knowledge to decide. Addressing Blind Guessing and LLMs May Perform MCQA by Selecting the Least Incorrect Option document this for multiple-choice answering. Neither study asks what in the input produces the uncertainty, and neither considers recommendation. Second, WikiQA shows that 20.3% of correct answer sentences share no content words with the question. That gives a realistic, naturally occurring zero-overlap subset on which the mechanism can be tested outside recommendation.

The closest existing work is the persona-compression line. PersonaX clusters long histories by item embedding, selects a core set of interactions, and generates cached textual personas. SIREN builds "Liked/Disliked Topics" sketches with K-means and appends recent items. Both target context length and the "lost-in-the-middle" problem, where content in the middle of a long prompt is ignored. Neither measures order sensitivity over candidates or persona traits, and neither asks whether persona wording matches candidate wording. The work closest on the order-bias side is Do RAG Systems Suffer From Positional Bias?. It studies order bias under retrieval and finds that moving relevant passages to the edges of the prompt also moves hard distractors there, so the benefits cancel. That study treats order bias as something to be managed through reordering. We treat it as partly caused by the text and test whether changing the text removes it.

Research Program

The pipeline starts from PersonaX-style personas, built by clustering histories and generating one textual trait per cluster, on Amazon Books, CDs and Vinyl, and the long-history Books480 split, plus SIREN's Amazon Movies setting. We add one ingredient, vocabulary-aligned persona rewriting. For each behavioral cluster we mine the terms that occur most distinctively in the metadata of the cluster's items, such as genre tags, descriptive phrases, and category paths. We then ask an LLM to restate the trait using those terms, under a constraint that the restatement's embedding similarity to the original trait, measured by a held-out encoder, stays above a threshold. Rewriting happens offline and is cached, so online cost is unchanged.

Order sensitivity is measured along two axes. One is the order of candidate items. The other, addressing the unstudied question raised by SIREN's layout, is the internal order of persona traits. For each user we sample k permutations along each axis. We define the per-instance sensitivity $S$ as the Jensen–Shannon divergence between the model's choice distributions under different permutations, following the measurement used in Serial Position Effects of Large Language Models. We also record the variance of the ground-truth item's rank.

The analysis then regresses $S$ on two quantities. The first is lexical overlap between candidate and persona, computed as content-word Jaccard overlap. The second is semantic similarity between them, computed as encoder cosine. The prediction is a negative coefficient on overlap that survives conditioning on similarity.

Correlation is not enough, so the core test is an intervention. On the same users we build three persona variants with matched meaning: the original, the catalog-aligned rewrite, and a deliberately misaligned rewrite. The misaligned version swaps catalog terms for synonyms absent from item text, under the same embedding-similarity constraint and a human-verified equivalence check on a sample. If the mechanism is real, $S$ should rank misaligned > original > aligned, and accuracy should move in the opposite direction. The theory in On the Emergence of Position Bias in Transformers explains the positional prior that is present regardless of content, through attention piling onto the first token in deep causal networks. Our claim is narrower and compatible with that theory: content that cannot be matched determines how much the decision leans on that prior.

Baselines

The naive reference is the PersonaX persona and the candidate list given to the recommender in a single fixed order with no debiasing. It anchors both accuracy and sensitivity. For this setting the file reports only relative figures: PersonaX "enhances AgentCF by 3–11% and Agent4Rec by 10–50%." It reports no absolute MRR@10 on Lacuna for PersonaX on Amazon Books or CDs and Vinyl, so we will reproduce that number ourselves. A raw-history variant that keeps the most recent items serves as the profile-free floor, because PersonaX reports it underperforming the clustered personas.

The strongest competitor on order robustness is permutation self-consistency, which runs the same prompt under k shuffled candidate orders and aggregates the votes. The survey Unifying Bias and Unfairness in Information Retrieval discusses it as the standard fix for order effects in LLM rankers. It is the right yardstick because it removes order dependence by construction. We run it with k ∈ {2, 4, 8} on the same model, users, and candidate sets. It counts as beaten in either of two ways: matched accuracy at equal compute, meaning our aligned persona with one pass against permutation averaging with one pass, or within one standard error of k = 4 accuracy at one-quarter the cost. We do not expect to beat k = 8 on accuracy. The relevant comparison is the accuracy–compute curve.

The strongest cheap competitor is OTTER. It applies an optimal-transport correction at inference time that pushes the model's predicted label distribution toward a target, such as a uniform spread over positions, without training. It reports an average 15.9% accuracy gain on text classification tasks. We apply it to the per-position choice probabilities, using the same unlabeled users to estimate the target. If OTTER alone matches our sensitivity reduction, the effect is a global position prior rather than a per-candidate lexical effect, and the core claim fails.

Three ablations isolate the key ingredient. The first, a rewrite without the vocabulary constraint, separates "an LLM rewrote the persona" from "the persona now uses catalog words." The second is the misaligned rewrite described above. The third is candidate-side bridging, which appends to each candidate a short phrase from the persona instead of changing the persona. It tests whether the side of the gap that gets closed matters.

Why not just…?

Why not just average over permutations? It works, and it is the real threat on accuracy. Our proposal does not claim to beat it at high k. The claim is that a cause in the input text can be removed offline, so that one pass approaches the robustness of several. The accuracy–compute curve settles this. If aligned personas add nothing on top of permutation averaging and give no benefit at k = 1, the idea is not useful.

Why not just calibrate outputs with OTTER or a similar method? Calibration corrects the average tilt toward certain positions across instances. It cannot know which candidates the model failed to match. The overlap regression and the misaligned-persona intervention distinguish the two explanations directly. Calibration should leave the dependence of $S$ on overlap unchanged, and lexical alignment should flatten it. If calibration flattens it too, we will report that.

Why not just tell the model to ignore order, or use chain-of-thought? Serial Position Effects of Large Language Models found that attention-steering prompts fail and that outputs cluster by architecture rather than by prompt. The Order Effect found that few-shot examples sometimes make order effects worse. Chain-of-thought reasoning does reduce serial-position effects, but SIREN reports that online chain-of-thought costs roughly 100× the latency of its internalized alternative. We include a chain-of-thought arm as an accuracy upper reference, not as a deployable competitor.

Why not skip the LLM and rank by embedding similarity? An embedding ranker cannot be affected by order, and it is a fair challenge. We run it on the same personas. If it matches the LLM recommender's accuracy, then order bias in this setting is mostly a cost of an unnecessary design choice, and that is a finding worth reporting. The file gives no reported head-to-head on these Amazon splits, so this comparison is new data either way.

Proposed Experiments and Decisive Checks

The main experiments run on Amazon Books, CDs and Vinyl, Books480, and Amazon Movies with open models such as Qwen 2.5 7B and a Llama-family model. We report MRR@10 and NDCG@10 alongside $S$ and rank variance, with 20 candidates per user, one ground-truth item, and hard negatives retrieved by embedding similarity. Hard negatives are needed because Do RAG Systems Suffer From Positional Bias? shows that semantically close distractors shape positional dynamics. That paper also gives the reference point for how small reordering gains can be: 71.00% for random shuffling against 71.73% for MaxRel, which places the most relevant passages at the prompt edges, on Qwen 2.5 7B with BGE+RR, a difference that was not significant.

A transfer check moves outside recommendation. On WikiQA we shuffle candidate answer sentences in a listwise selection prompt and stratify questions into those whose correct answer shares content words with the question and those in the 20.3% zero-overlap subset. The reported reference is CNN-Cnt at 0.6520 MAP. We also run the MSMARCO relevance setting of The Order Effect, which reports F1 drops of up to 12.24% for GPT-4o mini under shuffling, and test whether low-overlap passages account for a disproportionate share of that drop.

The decisive result is the three-way persona intervention. If misaligned personas raise $S$, aligned personas lower it relative to the original, and the gap persists after matching embedding similarity and after OTTER calibration, then the lexical gap causes part of the order bias. If $S$ does not move, order bias in these recommenders is independent of the wording of the content, and the right investment is in permutation averaging or architectural change rather than in persona construction.