Which Voting Rule Does a Language Model Run In Context?

When a language model is shown several rankings of the same items and asked for the consensus, it is doing rank aggregation. This happens routinely: a judge model reconciles its own sampled verdicts, and a listwise reranker merges evidence from several passages. The textbook consensus, the Kemeny ranking (the ordering that disagrees with the fewest pairwise preferences across all input lists), is NP-hard to compute. Simpler rules such as Borda count (sum each item's positions) are cheap but give different answers on some inputs. So a model aggregating "in context", from the prompt alone with no weight updates, must be doing something cheaper than Kemeny. Nobody has measured what that something is.

This note proposes to identify the implicit rule. The central tool is a clean geometric fact. Represent the pairwise majority margins between items as flows on a complete graph. The Hodge decomposition of Statistical ranking and combinatorial Hodge theory splits that flow into an acyclic "gradient" part and a cyclic "curl" part, the triangles like A beats B beats C beats A. On a complete comparison graph the gradient part orders items exactly as Borda does, and there is no harmonic component. So Borda and Kemeny can disagree only when curl is nonzero. Curl magnitude is therefore a controllable, cheaply computed dial that separates "easy" aggregation, where every sensible rule agrees, from the NP-hard regime where rules diverge. We use it to ask two things: which rule a frozen model's in-context behavior matches, and whether few-shot demonstrations can move it from one rule to another.

Core Claim

Our hypothesis is that frozen language models aggregate rankings in context as a score-based, Borda-like rule, not as a pairwise-majority rule like Kemeny or Copeland (rank by the number of head-to-head wins). If that holds, three things follow. First, the model's gap to the Kemeny optimum should be near zero on low-curl profiles and should grow with the curl fraction. Second, on profiles where a Condorcet winner (an item that beats every other item head-to-head) exists but is not the Borda winner, the model should pick the Borda winner. Third, few-shot demonstrations labeled by a different rule should shift behavior only when those demonstrations are drawn from rule-discriminating, high-curl instances; demonstrations drawn from typical instances carry no information about which rule is intended.

The contribution is threefold: a characterization of in-context aggregation, a benchmark whose difficulty axis is grounded in Hodge theory, and a test of whether the chosen rule is carried by a compact internal "task vector". A task vector is the intermediate-layer hidden state that In-Context Learning Creates Task Vectors showed summarizes a few-shot task and can be patched into a demonstration-free forward pass.

Relation to Prior Work

The closest work, Towards Foundation Models for Consensus Rank Aggregation, trains a dedicated encoder–decoder transformer with REINFORCE to minimize Kemeny distance. It shows transformers can learn good approximate Kemeny aggregation, but only after training specifically for it. It does not ask what general frozen models do, nor which axioms their output satisfies.

We use that paper in two roles. It motivates our work, because it suggests the architecture has the capacity. It also supplies our generators: its random, repeat, and "jiggling" base-ranking generators, and its Top-100 countries dataset.

The Geva et al. task-vector paper supplies our method. It is also a constraint: it succeeded on single-token tasks and left structured outputs open. We therefore make the winner, a single item token, our primary readout, and treat full permutations as secondary.

Several papers show that the choice of rule matters, which motivates the question. Designing Rules to Pick a Rule: Aggregation by Consistency shows that different datasets are best served by different rules and selects among rules by split-half agreement. Axioms for AI Alignment from Human Feedback proves that convex-loss score rules such as Bradley–Terry violate Pareto optimality and pairwise majority consistency. Soft Condorcet Optimization for Ranking of General Agents shows Elo lacks a Condorcet guarantee and gives a differentiable aggregator that has one.

If our hypothesis holds, in-context aggregation inherits the axiom violations Ge et al. prove for score rules. Those violations would sit inside every listwise judge, including the pipelines where On Cost-Effective LLM-as-a-Judge Improvement Techniques and A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth aggregate judge outputs. Both of those papers aggregate outside the model, by averaging or weighted Bradley–Terry. Our object is the aggregation performed inside a single forward pass.

Research Program

Profile generation. We construct preference profiles of $m$ voters over $n$ items, with $n$ from 4 to 20 so that exact Kemeny via integer programming stays cheap. Profiles come from the Jin et al. generators, from Mallows models with controlled dispersion, and from PrefLib election data. For each profile we compute the curl fraction:

$$\kappa = \frac{\lVert Y - \operatorname{grad} s^\ast \rVert^2}{\lVert Y \rVert^2},$$

where $Y$ is the antisymmetric matrix of normalized pairwise majority margins and $s^\ast$ is the least-squares (Borda-equivalent) potential. We stratify profiles by $\kappa$.

Rule-discriminating strata. We deliberately oversample instances where Borda, Copeland, and Kemeny disagree about the winner or the ordering. Random profiles almost never discriminate between rules, so aggregate accuracy alone would hide the phenomenon.

Behavioral rule identification. We present profiles to open-weight models from the LLaMA-7B/13B/30B, GPT-J, and Pythia families used by Geva et al., plus larger open models. Prompts are zero-shot and few-shot, and we fix a set of voter orderings and formats. On discriminating instances we record which candidate rule the output matches. Across instances we fit a mixture over rules, following the logic of Berker et al.'s consistency selection but applied to the model as the "rule".

Steerability. We give eight to sixteen demonstrations labeled by Kemeny, by Borda, or by Copeland. The demonstrations are drawn either from discriminating instances or from ordinary instances, and we measure the resulting shift in matched rule. We then extract the final-token hidden state at the intermediate layer that Geva et al. identify as a sweet spot, around layer 15 in LLaMA-7B, and scan neighboring layers. We patch it into demonstration-free queries and test whether the rule transfers. A conflict test mirrors Geva et al.'s task-dominance experiment: the prompt text says "Borda" while the patched vector comes from Kemeny demonstrations, and we check which rule governs the output.

Baselines

The naive reference point is Borda itself computed exactly. It is what the hypothesis predicts the model imitates. Any claim that the model does "better than Borda" on Kemeny gap must be measured against it on identical profiles.

The strongest existing method for the underlying objective is exact Kemeny by integer programming. At our item counts it is fast, so it defines zero gap. Copeland and Soft Condorcet Optimization serve as the Condorcet-consistent alternatives.

The Kemeny Transformer of Jin et al. is the strongest learned competitor. We will run it on the same profiles if weights are available; otherwise we report its published qualitative ordering against heuristics only. There is no reported result on Lacuna for its numerical Kemeny gap.

For the applied judge setting, the reference is Lail & Markham's criteria-plus-ensembling judge on RewardBench 2. They report 85.8% accuracy, +13.5 points over a single-call baseline, using arithmetic-mean aggregation. We compare mean aggregation, Kemeny aggregation, and in-context aggregation by the same judge over the identical set of $k=8$ sampled judgments per example, under the same model and sampling budget.

Two ablations isolate the key ingredients. The first uses demonstrations drawn from ordinary rather than discriminating instances; this tests whether informative demonstrations matter. The second shuffles voter and item order under a fixed profile; this tests whether "rule" is really position bias. Beating the naive reference means a Kemeny gap significantly below Borda's within high-$\kappa$ strata. Successful steering means the matched rule changes on held-out discriminating instances at rates well above those seen under the ordinary-demonstration ablation. Patching succeeds if it retains a substantial share of that shift, measured against Geva et al.'s 80–90% retention on simple tasks and their 10–20% query-only floor.

Why not just…?

Why not just run Borda or an integer-programming solver and skip the language model? For producing consensus rankings, that is the right answer, and we do not claim otherwise. The question here is diagnostic. Language models already aggregate inside judge and reranking prompts. If they implement a score rule, the Pareto and majority violations proved by Ge et al. are silently present in those pipelines. The RewardBench 2 comparison tests whether moving aggregation out of the model, to explicit Kemeny, is worth doing.

Why not just fit a Mallows or Bradley–Terry model to the outputs, without task vectors? Behavioral fitting is in fact our primary analysis. Task vectors add one falsifiable question: is the rule a compact, transferable internal state, or does it emerge only from surface pattern-matching? If patching fails, we report the behavioral result alone. That failure would itself be evidence on the open question Geva et al. raise about structured tasks.

Why not just prompt with chain-of-thought to compute Kemeny explicitly? This is a real threat to any accuracy claim. Written-out reasoning can execute Borda and small pairwise tables, and it may reach near-zero gap at small $n$. We include it as a condition and expect its cost to grow with the $n^2$ pairwise table. Our characterization concerns the single-pass behavior that judges actually use, and we report where chain-of-thought closes the gap as a function of $n$ and $\kappa$.

Why not attribute everything to position bias? Order sensitivity in in-context learning is well documented. The shuffle ablation tests this directly. If matched-rule rates collapse under permutation, the correct conclusion is that the model follows no stable rule, and we would report that.

Decisive Checks

The hypothesis is supported if three things hold. The Kemeny gap should rise monotonically with $\kappa$ while staying near Borda's gap. On Condorcet-winner-not-Borda-winner instances, the model should pick the Borda winner well above chance. This effect should survive voter and item shuffling.

The hypothesis is refuted if either of two things happens. The model might track Copeland or Kemeny on discriminating instances, which would mean it computes pairwise majorities in context. Or its errors might be uncorrelated with $\kappa$, which would mean its failures come from something other than cyclic structure.

The steering result is independent of the rule hypothesis and informative either way. If discriminating demonstrations shift the matched rule, the choice of aggregation rule is itself something in-context learning can set.