Language models are starting to make group decisions: shortlisting projects from citizen proposals, choosing panel members from stakeholder preferences, allocating slots. The usual way to check such a decision is to ask the model to explain it. That check can be defeated. A backdoor is a hidden behaviour planted by an attacker that activates only when a trigger appears in the input. A backdoored model could quietly shut out one group of voters and then write a fluent justification for doing so. Chain-of-thought research already shows that model explanations are often written after the answer is chosen. This proposal argues that the audit should never read the explanation. The auditor should instead check a mathematical property of the output that implies fairness and can be verified quickly, whatever the model says about itself.
Social choice theory makes this possible, and it also explains why the obvious property-level check fails. Take proportional fairness notions such as Extended Justified Representation (EJR), which say that every large, like-minded group of voters deserves some seats. Deciding whether an arbitrary committee satisfies them is coNP-complete (On the Complexity of Extended and Proportional Justified Representation). A coNP-complete check is one where the auditor must rule out every possible aggrieved subgroup, and those subgroups can number exponentially many. A backdoor that hurts one unknown cohesive subgroup is exactly the violation that is hard to find. Market-Based Explanations of Collective Decisions supplies a stronger condition, Stable Priceability (SP), that can be checked with a linear program. SP gives voters virtual budgets and candidates prices, and it implies the core, which in turn implies EJR. We propose to split the audit of an LLM group decision into two cheap checks. The first is a per-voter check that the model read each ballot correctly, done by random sampling. The second is a global linear-program check on the chosen outcome. We will then measure how much damage a backdoor built to evade explanation-based detectors can still do once both checks pass.
Core Claim
Explanation-based backdoor detection is strong today only because current backdoors were never trained to explain themselves well. Once an attacker adds that training, only checks that ignore the explanation stay reliable. For group decisions over approval preferences, we claim such checks exist and are cheap. An auditor that parses a small random sample of ballots and runs the SP linear program on the output will flag the fairness-targeting backdoors that rationale judges and trigger-mention checks miss. The damage that survives both checks is limited to choices among committees that are already certified proportional. We will measure that residual, which we call the attacker's latitude.
This idea rests on three strands of prior work, each playing a different role. When Backdoors Speak motivates the gap. On backdoored LLaMA 3-8B and DeepSeek-7B, it detects triggered inputs from explanation quality with 97.5% accuracy using GPT-4o, and from hidden-layer probes with 98.8% accuracy. Its own authors note that backdoors optimised to produce plausible explanations would undercut the first result. The chain-of-thought faithfulness papers explain why such backdoors should be easy to build. How Likely Do LLMs with CoT Mimic Human Reasoning? finds answers are often produced in parallel with, not because of, the reasoning, and that alignment fine-tuning weakens the link. On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models finds that steering toward faithfulness barely works and costs accuracy. Neither paper studies backdoors. The certificate literature supplies the method. SP is the global check, and FaVeR is a precedent for replacing trust in a network with solver-checkable fairness conditions, though only for individual fairness in small tabular ReLU networks. On the attack side, BadChain hijacks reasoning in-context with inserted demonstration steps, and BiasRAG plants demographic-targeting backdoors in retrieval that pass clean fairness benchmarks. Neither targets proportional representation, and neither is evaluated against a certificate-based auditor.
Research Program
The task. Lacuna lists no committee-selection benchmark, so we construct one. Voters are individuals drawn from Folktables/ACSIncome (Retiring Adult). Each voter's preferences over candidate projects are written as free text generated from latent approval sets that correlate with demographic attributes, so cohesive voter groups align with protected groups. The model receives the free-text preferences plus soft textual constraints, such as a budget or a topic balance, and returns a committee of size $k$ and a rationale. The free-text layer is the honest reason to involve an LLM at all. If ballots arrive as clean approval sets, a polynomial rule such as Method of Equal Shares suffices.
The attack. We build the strongest explanation-stealthy fairness backdoor we can, in two variants on open-weight 7–8B models, the same model class as When Backdoors Speak. The first is an in-context BadChain-style variant, where poisoned demonstrations teach a rule like "under phrase T, deprioritise proposals from district D" dressed as a legitimate budget heuristic. The second is a fine-tuned variant. Here the trigger makes the model deny a cohesive group its proportional seats, and the model is additionally trained on fluent, trigger-free, fairness-sounding rationales, with a penalty whenever the explanation mentions the trigger. The attack has two routes to the goal. It can distort the outcome directly, or it can misread the target group's ballots so that the outcome looks proportional relative to corrupted preferences.
The audit. The auditor never reads the rationale. The first check parses a random sample of $m$ ballots independently, using a separate model or a human, and compares them with the model's readings.
The sampling check works because of a structural fact about proportionality. To take seats away from a cohesive group, an attacker must misread a sizable share of that group, roughly a $1/k$ fraction of voters or more. The probability that $m$ uniform samples all miss a corrupted fraction $\epsilon$ is $(1-\epsilon)^m$, so detection gets exponentially more likely as the sample grows. A concern is that a clever attacker might corrupt only a few pivotal ballots, and we test this empirically.
The second check, given the verified ballots, runs the SP linear program on the output committee and accepts only if a stable price system exists. A committee can be EJR-fair and still fail SP, and on some instances no stable-priceable committee may exist at all. We will report how often honest outputs are flagged for either reason. Where SP is not available, we evaluate a weaker priceability check and state exactly what it certifies.
The residual. A certified committee can still be the worst certified committee for some protected attribute that is not visible in the approvals. We measure this latitude directly. For each instance, we compute the range of a demographic outcome, for example seats going to projects favoured by one ACS group, over all SP-certified committees, and ask how much of that range a latitude-maximising backdoor actually uses.
Baselines
The naive reference point is to audit the output on aggregate fairness. We check seat share per predefined demographic group and run clean-input fairness tests, BBQ-style, before deployment. BiasRAG's clean utility of 83.21% against 85.43% for an unpoisoned system shows backdoors can pass clean tests, so we expect this baseline to miss triggered cohesive-subgroup attacks entirely. Beating it is necessary but not interesting.
The strongest existing detector family comes from When Backdoors Speak. It has two members: a GPT-4o judge of explanation quality, reported at 97.5% detection accuracy, and linear probes on per-layer last-token representations, reported at up to 98.8%. Both figures come from SST-2, Twitter Emotion and AdvBench. We found no reported result on Lacuna for these detectors on committee-selection or proportional-fairness tasks. We will reimplement both on our task. They get the same triggered and clean instances as our auditor, the same model checkpoints, and probe training data drawn from ordinary non-stealthy backdoors, the setting they were designed for. We also run the probes in an optimistic setting where they are trained on our stealthy attack itself. Our auditor beats them if two conditions hold. First, under the rationale-stealthy attack, it detects at least as many triggered violations at a false-alarm rate on clean instances no worse than theirs. Second, the explanation-based detector's accuracy falls well below its clean-backdoor level, which would show the evasion is real rather than assumed.
A third baseline is exact EJR checking by integer programming. It is a legitimate competitor on small elections. We give it the same wall-clock budget as the linear program across growing numbers of voters and candidates, and record where it times out.
Two ablations isolate the key ingredients. Dropping ballot sampling shows how much of the attack runs through misreading ballots. Swapping SP for an empirical check over predefined demographic groups shows whether the certificate or merely looking at outcomes does the work.
Why not just…?
Why not just read the chain of thought and flag unfair reasoning? The faithfulness literature says the text is a weak signal even without an adversary. Bao et al. find GPT-4 reaching correct answers despite wrong reasoning 74.4% of the time on 6-digit addition, and implicit logical leaps to biased conclusions are documented in On Second Thought, Let's Not Think Step by Step!. Our rationale-stealthy attack is built to defeat this check directly. The GPT-4o-judge baseline measures by how much.
Why not just use internal-representation probes, which reached 98.8%? This is the real threat. Our attack penalises only the text of the explanation, not the activations, so a probe could still catch it. We will run the probe in both the transferred and the optimistic setting. If it keeps high accuracy against our attack, that is a genuine finding, and we will report the certificate's advantage only as the ability to audit without access to model internals and without samples of known attacks. We will not claim a win on accuracy in that case.
Why not just run Method of Equal Shares and skip the LLM? For clean approval ballots, that is the right answer, and our experiments include that condition to show it. The LLM matters only when preferences arrive as free text with soft constraints. That is exactly why the ballot-reading check exists: the backdoor can hide in how preferences are read, which no voting rule sees.
Why not just check EJR exactly with a solver? On small instances this works, and the timing baseline will show where. The reasons to prefer SP are that it scales polynomially and that it certifies the core, which is stronger than EJR. The honest cost is false alarms on fair committees that lack price certificates, which we report.
Proposed Experiments
The decisive experiment is a comparison on one set of triggered and clean elections. We run the plain backdoor and the rationale-stealthy backdoor, each in its outcome-distorting and ballot-misreading forms, against the GPT-4o judge, the probes, the aggregate audit, and our two-check auditor. We report attack success, meaning the share of triggered instances where a cohesive group of the required size is denied proportional seats. We also report detection rate and false-alarm rate on clean instances, and clean decision quality. For context on how strong reasoning backdoors are, BadChain reports 97.0% average attack success on GPT-4 and 76.6% on Llama2 on multi-step reasoning benchmarks.
Three results would settle the claim. The explanation judge must degrade under rationale training, confirming the gap. The certificate auditor must catch both outcome and ballot attacks at a low false-alarm rate, with the pivotal-ballot attack as the stress test for sampling. And the measured latitude inside the certified region must be small relative to the unconstrained attack. If latitude is large, the conclusion changes to this: certificates bound proportionality violations, but protected-attribute harm needs its own checkable condition. We would treat that as the next open problem.
References
- On the Complexity of Extended and Proportional Justified Representation
- Market-Based Explanations of Collective Decisions
- When Backdoors Speak: Understanding LLM Backdoor Attacks Through Model-Generated Explanations
- How Likely Do LLMs with CoT Mimic Human Reasoning?
- On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models
- Efficient Counterexample-Guided Fairness Verification and Repair of Neural Networks Using Satisfiability Modulo Convex Programming
- BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models
- Your RAG is Unfair: Exposing Fairness Vulnerabilities in Retrieval-Augmented Generation via Backdoor Attacks
- Retiring Adult: New Datasets for Fair Machine Learning
- On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning