Refuting Voting Rules as a Reinforcement Learning Curriculum for Logical Transfer

Language models trained with reinforcement learning on problems whose answers a program can check get better at math, and some of that gain carries over to code, science questions and logic puzzles. Nearly all of these checkable problems share one shape: compute a single answer and compare it to the key. Much of logical reasoning has a different shape. It asks whether a general claim holds for every case, and if not, it asks for the one case that breaks it. Social choice theory, the mathematics of voting rules, is full of exactly these claims. "Borda count is monotone" and "Instant runoff satisfies participation" are each either true or refuted by a small election, and for elections with a handful of voters and candidates a computer can confirm any proposed counterexample by brute force.

We propose using this as a training domain. A model is rewarded for deciding whether a voting rule satisfies a stated property (an axiom) and, when it does not, for producing a concrete election that a checker confirms as a violation. The question is whether this kind of training transfers to deductive and causal reasoning benchmarks better than math training does under the same compute. We separate the contribution of the task's shape (find a counterexample) from its content (social choice) with a controlled two-by-two design. The design is aimed at a skeptic's most natural objection, that any counterexample task would work just as well.

Core Claim

Reinforcement learning with verifiable rewards (RLVR: the model samples solutions and is rewarded only when an automatic checker accepts them) has so far mostly been run on answer-computing tasks. Our claim is that tasks requiring a checked witness, which means refuting a universal statement by building a counterexample, train a different and more transferable skill. That skill is quantifier reasoning: keeping "for all" and "there exists" distinct, and searching structured case spaces instead of pattern-matching surface text. The concrete prediction is that, at equal compute and on the same base model, training on social choice witness tasks raises accuracy on FOLIO and on the perturbed variants of Corr2Cause more than math RLVR does, without losing much on MATH500.

The secondary claim is that social choice adds value beyond the witness format. Its axioms are relational, its objects come with natural-language renderings, and its characterization theorems supply ground truth. Uniqueness results determine which rule a set of axioms forces, and impossibility results guarantee that some violation always exists. A propositional-logic counterexample task of matched difficulty is the control for this second claim. If that control transfers just as well, we will report that the gain comes from the task shape and not from voting theory.

Where this sits

The axiomatic literature supplies both content and ground truth. Characterizations such as Minimax as an extension of May's theorem (Holliday and Pacuit), Split Cycle's uniqueness under coherent independence and positive involvement (Ding, Holliday, and Pacuit), and Thiele and proportional approval rules (Lederer et al.) each state that a fixed set of axioms pins down one rule. Those theorems become tasks with known answers. Impossibility results, such as the sequential-voting dichotomy of Xia et al., guarantee that any candidate rule violates something, so a witness always exists. Troquard, van der Hoek, and Wooldridge show that axiom satisfaction on finite domains is decidable but grows exponentially with voters and candidates. That result both justifies the checker and constrains it: we stay at sizes of roughly five candidates and nine voters, where enumeration is cheap.

Three bridging papers motivate the work and show what is missing. Lanctot et al. use voting rules to rank agents after the fact. Ge et al. prove that standard reward models violate Pareto optimality and majority consistency, and they propose an axiom-respecting way to aggregate preferences. Both use social choice as machinery around the model. We instead use it as material the model reasons about. Tak et al. audit whether reasoning models obey rational-choice axioms but do not train on them, so their test suite serves us as a near-domain evaluation.

The closest prior work is Vashishtha et al.. They fine-tune transformers on demonstrations of causal axioms, transitivity and d-separation, presented as premise–hypothesis–conclusion tuples, and report a 20 percentage point F1 gain on Corr2Cause for Llama-3-8B. Our proposal differs in three ways. It uses reinforcement learning with checked witnesses rather than supervised imitation of conclusions. It uses voting axioms rather than causal ones. And it treats the causal benchmark as a transfer target rather than the training domain. Their data also serves as one of our baselines.

On the transfer side, 1-shot RLVR, Reasoning Vectors and CPL establish that narrow math RL transfers. Jin et al. constrain the design: models that appear to have learned formal rules collapse when variable names or wording change. That finding is why every task we train on is rendered with randomized names and formats, and why the perturbed splits are our primary metric. Ota et al. show that an agent trained only inside a bare propositional axiom system produces lemmas that help other models. We borrow that setting to build the propositional control.

Research Program

We will build a task generator with a brute-force checker. The rule pool covers plurality, Borda, Copeland, Minimax, Schulze, Split Cycle, instant runoff, approval voting and proportional approval voting. The axiom pool covers anonymity, neutrality, monotonicity, Pareto, Condorcet consistency, independence of irrelevant alternatives, participation, positive involvement, reinforcement and clone independence.

Each instance has one of three forms. In the first, the model decides whether a rule satisfies an axiom and, if it says no, supplies a violating election (or pair of elections for axioms that compare two). In the second, the model is given only the axioms of a uniqueness theorem and a profile, and it must output the outcome those axioms force. In the third, the model is given an incompatible axiom set and a proposed rule, and it must exhibit a violation.

The model's output is parsed as structured text. The reward is one only if the verdict is correct and any witness passes enumeration. This rules out the coin-flip reward hacking that a bare yes/no verdict would allow.

Every instance is rendered in several surface forms: ranking tables, prose narratives about committees or juries, and randomized candidate names and orderings. The goal is that no surface template predicts the label. We hold out some axioms (positive involvement, clone independence) and the whole family of approval-based committee rules to measure within-domain generalization.

We will train Qwen2.5-Math-1.5B and Qwen2.5-7B, the base models used by 1-shot RLVR and Reasoning Vectors, with Group Relative Policy Optimization (GRPO: a policy-gradient method that scores each sample against others drawn for the same prompt). All four cells of the design get the same rollout budget, hyperparameter search and number of gradient steps. The four cells are:

  • social choice witness tasks
  • social choice answer-only tasks: the same instances, but the model only outputs the forced winner or a yes/no verdict, with no witness
  • math RLVR
  • propositional-logic witness tasks: refute a claimed tautology with a falsifying assignment, or exhibit a derivation, in a Hilbert-style system like Ota et al.'s

The social choice and propositional witness tasks are calibrated so that the base model succeeds at similar rates.

Baselines

The naive reference point is the untrained base model, prompted with chain of thought on every evaluation. Every gain is measured against it, and its accuracy on our own social choice test set tells us whether the domain is hard enough to train on at all.

The strongest competitor on the transfer question is compute-matched math RLVR on the same base model. Math training is the established route to reasoning transfer, and it is what a reviewer will say we should simply do. We run it ourselves with the same GRPO settings, rollout budget and tuning budget as the social choice runs. As published anchors, 1-shot RLVR reports MATH500 accuracy rising from 36.0% to 73.6% for Qwen2.5-Math-1.5B. Reasoning Vectors reports a +12.3% Big-Bench Hard (BBH) gain for Qwen2.5-1.5B from a math-GRPO weight difference, the clearest reported evidence that math RL transfers to logic. We found no reported result on Lacuna for math RLVR on FOLIO or on Corr2Cause's perturbed splits, so our own runs supply that number. Beating this baseline means a statistically significant improvement over the math-RLVR model on FOLIO and on Corr2Cause's variable-refactored and paraphrased splits, a BBH gain at least matching math RLVR's, and a MATH500 drop of no more than a few points.

The strongest supervised competitor on the causal target is axiomatic fine-tuning. We use the causal-axiom data of Vashishtha et al., which reports a 20 percentage point F1 gain on Corr2Cause for Llama-3-8B, applied to our base models with matched tokens. Because their training domain is causal and ours is not, merely approaching their Corr2Cause gain while beating them on FOLIO would already be informative. We also include supervised fine-tuning on solver-generated solutions to our own social choice tasks, which separates reinforcement learning from the content itself.

The two ablations that isolate the key ingredients are the answer-only social choice cell, which tests whether the witness requirement matters, and the propositional witness cell, which tests whether social choice content matters. For frontier context, Jin et al. report zero-shot GPT-4 at 29.08% F1 on Corr2Cause. They also report fine-tuned RoBERTa-Large falling from 94.74% to 67.87% under variable refactoring and to 55.45% under paraphrase. FOLIO reports roughly 70% for few-shot GPT-4, roughly 77–78% for solver-assisted pipelines, 65.7% for fine-tuned Flan-T5-Large, and 53.10% for GPT-4 on the deeper HybLogic subset.

Why not just…?

Why not just run more math RLVR? Math RLVR is the real threat, which is why it is the main matched baseline rather than a straw man. The evidence that it transfers to logic is the +12.3% BBH figure from Reasoning Vectors. However, math rewards reward computing one value, and nothing in that signal penalizes confusing "some case works" with "every case works." If the math cell matches or beats the social choice witness cell on FOLIO and the perturbed Corr2Cause splits, our central claim is false, and the experiment is designed to show that plainly.

Why not just use any counterexample task, like propositional logic? This alternative could plausibly work as well as our proposal, so we made it a cell of the design instead of arguing against it. If the propositional witness cell transfers as well as the social choice cell, the paper's claim becomes "witness-construction rewards transfer better than answer rewards," and social choice is one convenient source among several. We expect an edge for social choice on FOLIO-style problems and the rational-choice audit for two reasons. Its relational, multi-entity structure and natural-language renderings resemble those problems more closely than bare formulas do. And its theorems provide graded, guaranteed-solvable impossibility tasks. Whether that expectation holds is an empirical question the design answers.

Why not just fine-tune on axiomatic demonstrations, as Vashishtha et al. did? Supervised imitation teaches the model to reproduce the conclusions of axioms. Jin et al.'s collapse under refactoring shows that imitation can latch onto surface templates. Our witness reward cannot be satisfied by a template, because the checker inspects the election the model actually constructs. The supervised social choice cell and the transplanted causal-axiom baseline directly measure whether reinforcement learning with witnesses buys robustness that imitation does not.

Why not just prompt a frontier model or call a solver? The file's numbers argue against both as complete answers. GPT-4's 29.08% on Corr2Cause and 53.10% on HybLogic show the limits of prompting. Solver pipelines plateau near 77–78% on FOLIO because translating natural language into first-order logic succeeds only 65.21% of the time for GPT-4. Both serve as reference points, but neither tells us what training on a small open model can achieve.

Proposed Experiments

The decisive check is the interaction in the two-by-two design. Our claim requires the social choice witness cell to beat both the answer-only cell and the math cell on FOLIO accuracy, on its HybLogic subset, and on Corr2Cause F1 under variable refactoring and paraphrase. Robustness is measured as the gap between original and perturbed splits, with Jin et al.'s RoBERTa drops as the reference for what brittle learning looks like. Big-Bench Hard logic subsets and GPQA broaden the transfer picture. MATH500 checks that we do not trade away math, and the rational-choice axiom suite of Tak et al. serves as a near-domain test with no reported numerical baseline on Lacuna.

Within the domain, accuracy on held-out axioms and on approval-based committee rules shows whether the model learned to search for counterexamples or memorized specific rule–axiom pairs. We will also inspect a sample of generated witnesses: a model that has learned the skill should build minimal violating elections rather than oversized lucky ones.

Each configuration runs with three seeds, and we report confidence intervals. A null result here would still be useful: it would show that the reward's structure does not matter and that transfer depends only on the generic pressure of reinforcement learning.