Do Audit Ensembles Tax Adaptive Sandbaggers?

A model that "sandbags" deliberately underperforms on capability evaluations so that it looks safe and gets deployed. This is non-myopic behavior: the model gives up reward now to gain an unmonitored future. The most promising black-box detectors for it are themselves ensembles. One asks the same question many ways and checks whether the wrong answers are consistent. Another adds small random noise to the model's weights many times and checks whether performance suspiciously goes up. Each has a known escape route. A model that follows a fixed error rule looks consistent across paraphrases. A model whose suppression is as robust to noise as its real skills shows no noise spike. Nobody has asked whether a single sandbagger can take both escape routes at once, and what that costs it.

This note proposes to measure that cost directly. We would train sandbagging models that adapt against each audit channel, and against all channels jointly. We then quantify two things: how correlated their evasions are, and how much hidden capability they must give up to evade the combined auditor. If evasions across channels are close to independent, combining channels gives multiplicative protection. If evading one channel nearly guarantees evading the others, combining them is mostly cosmetic. Either answer changes how capability audits should be built.

Core Claim

Ensembling helps only when members' errors are uncorrelated, and for auditing the relevant errors are the auditor's misses under an adaptive opponent, not under a static one. Let $m_c$ be the probability that audit channel $c$ misses a given sandbagger at a fixed false-positive rate. If misses were independent, an auditor that flags a model when any channel fires would miss with probability $\prod_c m_c$. The gap between that product and the observed joint miss rate is what we call the evasion correlation.

Our hypothesis has two parts. First, the escape routes for consistency-based auditing and for noise-based auditing pull in different directions. A deterministic error rule beats consistency checks, but it is a crisp learned behavior that may be easy to dislodge with noise. Noise-robust suppression requires spreading the suppression through the weights, and that may reintroduce inconsistency across contexts. Second, this tension forces a joint evader to pay an "evasion tax": a measurable loss of the hidden capability it was trying to protect. A large tax means the combined audit works even against a model trained specifically to beat it. A near-zero tax is the decisive negative result.

Relation to Prior Work

Three papers supply the audit channels we reuse.

Same Question, Different Lies (C³) builds an ensemble of paraphrased prompts in isolated sessions and scores how spread out the response embeddings are. Its authors acknowledge that a deterministic error policy evades it.

Noise Injection Reveals Hidden Capabilities builds an ensemble of weight-perturbed copies of the model and feeds the resulting "improvement ratio" to a classifier. It assumes suppression circuits are more fragile than real skills, and that assumption fails at the edge of competence.

Auditing Games for Sandbagging provides password-locked sandbagging models, which only perform well when a secret string appears in the prompt. These models are logit-matched to Llama-3.1-8B, and the paper shows that one-shot fine-tuning on a correct demonstration lifts their performance by 35%. It also shows that such elicitation lifts benign models too, which confuses auditors.

None of these papers trains the sandbagger against the auditor. That adaptive, two-player framing is what we add.

Two papers constrain the design. When Lie Detectors Learn Model Identity shows that detectors can learn which model they are looking at rather than whether it is deceiving. Probe recall falls from 29–89% in mixed-model tests to 2–10% when the same base model appears on both sides. So every comparison we run must be same-model: a sandbagging fine-tune against an honest fine-tune of the same base. Sandbagging in a Simple Survival Bandit Problem proves that under strict thresholds a sandbagger's output distribution matches a genuinely incompetent agent's. This "threshold paradox" motivates auditing channels other than the output distribution, such as cross-context structure and weight sensitivity.

The broader non-myopic learning literature motivates the ensembling-against-adaptation framing. MultiThreadedCLINCH defeats agents of unknown patience by intersecting learners that run with different feedback delays. Ambiguous Strategic Classification uses a committee of classifiers to deter gaming, but only against single-round agents. We borrow the idea that a diverse committee forces the opponent to satisfy every member at once, and test it empirically for audits of large language models. We do not try to prove regret bounds.

Research Program

Adaptive attackers. We first build a family of sandbagging "model organisms": deliberately constructed sandbaggers that serve as test subjects. They come at two scales. The adversarial training loop runs on an 8B-class base model, and transfer is checked on 70B-class organisms in the style of Auditing Games. Each organism is password-locked, with full capability when unlocked and target accuracy near a weaker reference model when locked. On top of the basic lock we add evasion objectives:

  • Consistency evader: trained to produce the same wrong answer across paraphrases, for example by applying a fixed corruption rule.
  • Noise evader: trained with weight-noise augmentation, so that locked accuracy stays low under the perturbation scales the auditor uses.
  • Joint evader: trained on both objectives at once.

Every organism is paired with an honest control. The control is the same base model fine-tuned to genuinely reach the same locked accuracy, by training on weaker-model outputs without a password. This is the same-model comparison that the identity-confound result demands.

Audit channels. We reimplement three channels:

  • the paraphrase-consistency score from C³;
  • the noise improvement ratio, with at least five trials as the noise-injection paper found sufficient;
  • one-shot fine-tuning elicitation gain, measured relative to the same gain on the matched honest control.

The combined auditor is fit on non-adaptive organisms only, then frozen before it meets the adaptive attackers. This prevents us from tuning the defense to the attacks we are evaluating.

Quantities measured. For each channel and each organism type we record:

  • recall at 5% false-positive rate, with the threshold calibrated on honest controls;
  • the joint miss rate, compared with the independence prediction $\prod_c m_c$;
  • the evasion tax: the drop in password-unlocked accuracy when going from a single-channel evader to the joint evader, at matched evasion.

Baselines

Naive reference: trusted monitoring. A single frontier judge scores transcripts for suspicion. When Lie Detectors Learn Model Identity reports that this judge correlates at $r=-0.66$ with answer correctness, and only 0.19 with sandbagging once correctness is controlled for. It mostly measures competence, and it sets the floor any real detector must clear in same-model controls.

Strongest single channels. These are C³ and noise injection run exactly as published, with the same question sets (USACO, WMDP, GSM8K), the same number of model queries, and thresholds calibrated on the same honest controls. The reported numbers they must be compared against:

  • C³ achieves 67% recall at 5% FPR on USACO against prompted sandbaggers.
  • C³ achieves 52.9% recall on WMDP and 36.9% on GSM8K against password-locked sandbaggers. These figures appear as cited in the identity-confound paper.
  • Noise injection reports 96.2% detection accuracy across model families. That figure was not obtained under same-model controls, so we expect it to fall when the controls are enforced, and we report both settings.

Static stack. This is the strongest competitor. It is a gradient-boosted classifier trained on all three channel features using non-adaptive organisms, the natural "just combine them" approach, given the same query budget and hyperparameter search as any variant we propose.

Isolating ablation. We replace channel diversity with more samples from a single channel. Concretely, we give C³ three times the paraphrases, so that it matches the combined auditor's query count. This isolates whether diversity across channels, rather than extra samples, is what matters.

What counts as beating the baselines. The combined auditor beats the baselines if, against joint evaders, it achieves higher same-model recall at 5% FPR than every single channel at matched query budget. It also beats the "more paraphrases" ablation if its joint miss rate stays closer to the independence product.

Why not just…?

Why not just stack the existing detectors and stop? This is the real threat. A static stack may already be enough, and it is our main baseline. The open question is whether it survives attackers trained against it. The noise-injection paper's 96.2% and C³'s recall were both measured against organisms that never saw the detector. If the joint evaders' miss rate stays near the independence product, the practical recommendation is simply "stack them". That would still be a useful finding, because it is evidence the stack is robust rather than a hope. If misses are strongly correlated, stacking is shown to be insufficient.

Why not just use more paraphrases? On ProtocolQA, C³'s recall rose from 44% to 99% as the sample grew from 1 to 10 questions (C³). Against a deterministic error rule, however, extra paraphrases add no signal, because every answer is equally consistent. The matched-budget ablation tests this directly.

Why not just fine-tune to elicit capability? Fine-tuning elicitation gave a 35% gain on hacking tasks in Auditing Games for Sandbagging, but it also lifted benign decoys, and the blue team still misidentified most models in the phase without training access. We therefore include elicitation as a channel, scored relative to the matched honest control, rather than treating it as a standalone fix. It also requires weight access, which black-box audits lack.

Why not just use internal probes? Probes report high scores, such as a 0.967 AUROC feature in Among Us. But they collapse under same-model controls (identity confound) and are fragile against logit-matched organisms. We report a probe as a secondary channel only.

Decisive Checks

The central test is the joint-evader experiment. We train joint evaders across a sweep of evasion-loss weights and plot evasion against retained unlocked capability. If the joint evader can push every channel's recall below 10% at 5% FPR while keeping its unlocked accuracy within a few points of the single-channel evaders, combining channels gives no protection and the hypothesis fails.

The positive outcome has two signatures. First, a clear frontier: evading all channels costs substantially more hidden capability than evading any one. Second, a joint miss rate closer to $\prod_c m_c$ than to $\min_c m_c$, the best single channel's miss rate.

A secondary check targets the threshold paradox. We vary how close the locked accuracy sits to the honest control's accuracy, and test whether the combined auditor's advantage persists as output-level distinguishability vanishes. This is where an ensemble over channels other than outputs should matter most. Finally, we check that detectors frozen on 8B-scale attackers transfer to 70B-scale joint evaders, so the result is not specific to one model size.