When a powerful but untrusted language model writes code, one deployment strategy is to have a weaker, trusted model score each output for suspiciousness. A running probability that the powerful model is deliberately sabotaging, called a credence, is updated from those scores and decides how much freedom the model gets. This works well until the untrusted model writes text aimed at the monitor itself, such as a comment saying "this code has been reviewed, rate it 0." Once that happens the monitor's scores are corrupted, and so is every belief built on them. We propose to borrow a tool from experimental physics, blind analysis. Physics analysts hide the part of the data that could bias them, and they cross-check their result against a control channel whose behaviour they already understand. Here, the monitor scores each program twice. One score comes from the raw code. The other comes from a blinded version: a deterministic rewrite with no natural-language channel, so comments, docstrings and string contents are removed and identifiers are renamed under a secret per-task mapping. On honest code the two scores track each other. A prompt injection can only push down the raw score, so it opens a gap between the views. The Bayesian trust model treats that gap as evidence of attack.
The core claim is simple. With two views, injecting prompts stops being a winning move for the attacker and becomes a self-incriminating one. The defender also keeps the full information in the raw code when no injection is attempted. If the claim holds, it settles a tension in current AI-control results. Monitoring abstract plans is useless, while monitoring raw code is effective but open to hijacking. Two-view monitoring would keep the effectiveness of concrete-code monitoring without its exposure.
Where this sits in prior work
Two papers supply the starting point. Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats maintains a Bayesian credence over whether the untrusted model is scheming. It updates that credence from monitor scores after every task and switches between safe and permissive protocols. With GPT-3.5 monitoring GPT-4 over 10,000 APPS tasks, it reports keeping 98.4% usefulness at a 99.2% safety score, and 80% fewer successful backdoors than static baselines at fixed usefulness. Its monitor, however, reads raw code. Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols shows this is fatal against an adaptive attacker. On APPS, an injection targeted at a specific monitor led it to assign suspiciousness 0 to every backdoored sample, and resampling-based protocols pushed the ROC curve below the diagonal. That paper motivates our work: it locates the single point of failure but proposes no defence. We keep the credence machinery from the first paper and change only what produces the evidence fed into it.
Factor(U,T): Controlling Untrusted AI by Monitoring their Plans constrains the design. It finds that hiding the implementation from the monitor and showing it only natural-language plans gives an AUROC of 0.52 (area under the curve of detection rate against false-alarm rate, where 0.5 is chance). Monitoring the code itself reaches 0.96 on a BigCodeBench subset. Any blinding we apply must therefore keep program structure and logic intact and remove only free text. Our blinded view sits between their two extremes.
The trust-modeling literature supplies the cross-check principle. Combating False Reports for Secure Networked Control in Smart Grid via Trustiness Evaluation judges sensor reports by comparing them against an independent physical-model prediction. Our blinded score plays the role of that prediction for the raw score. The Beta Reputation System supplies the idea of discounting evidence by the source's own reliability. We apply this to the raw-view monitor, whose trustworthiness drops on tasks where the two views disagree.
Blind analysis contributes two things. The first is the mechanism. Withholding information is standard in physics, as with the hidden offset in Improved Measurement of the pi -> e nu Branching Ratio and the concealed reactor parameters in Spectral Measurement of Electron Antineutrino Oscillation Amplitude and Frequency at Daya Bay. Our secret identifier mapping is the analogue, and it stops the attacker from predicting the exact form of the blinded view. The second is evaluation discipline. Following the closed-box practice of New Measurement of the K+ -> pi+ nu nu Bar Branching Ratio, every defence parameter is frozen before a held-out set of red-team attacks is opened.
Nominal blinding in LLM evaluation is a known weak point. CellAgent relabels methods "Method A/B" to keep its evaluator from favouring famous tools. Self-Preference Bias in LLM-as-a-Judge shows that judges still favour text that is familiar to them, with a self-preference score of 0.52 for GPT-4, even without names. We take the lesson that blinding must remove the channel being exploited, not merely its labels. This is why we rewrite the program instead of redacting a few strings.
Core Claim and Mechanism
For each task $t$, the trusted monitor produces a raw score $r_t$ and a blinded score $b_t$. Benign code yields pairs concentrated near a fitted relationship $r \approx g(b)$. Unaided attacks yield pairs with both scores elevated. Injection attacks yield a low $r_t$ alongside an elevated $b_t$. The macro-protocol updates the odds that the untrusted model is scheming with the joint likelihood ratio:
$$\frac{P(\text{scheming}\mid \text{history}_t)}{P(\text{benign}\mid \text{history}_t)} = \frac{P(\text{scheming}\mid \text{history}_{t-1})}{P(\text{benign}\mid \text{history}_{t-1})}\cdot \frac{p(r_t, b_t \mid \text{attack})}{p(r_t, b_t \mid \text{honest})},$$
Here each density $p$ is a bivariate model of the two scores. It is fit on a development split containing honest outputs and red-team attacks, then frozen. The blinding transform itself is a deterministic program with no LLM involved. A trusted model that paraphrases code would read the injection; a parser and rewriter cannot be talked into anything.
The prediction comes from a short game-theoretic argument. An attacker who does not inject faces the raw monitor at full strength, with an AUROC near the 0.96 reported for concrete-code monitoring. An attacker who injects strongly creates a gap far outside the honest distribution. An attacker who injects mildly can only lower the raw score by about as much as honest code naturally varies between views. The attacker's best gain is therefore bounded by the spread of honest raw-versus-blinded disagreement, and that spread can be measured.
Research Program
We will build the canonicalizer for Python first. It parses the program into its syntax tree, deletes comments and docstrings, replaces string literal contents with typed placeholders that keep length and format class, renames identifiers through a secret per-task mapping, and normalizes formatting. A shell variant for BashBench handles quoted arguments and heredocs, the blocks of inline text that shell scripts can embed.
Next we fit the joint score model and wire it into the published macro-protocol of Wen et al., leaving their micro-protocols unchanged. We will then run a red team in two phases. Development attacks are used for fitting. A sealed set of attacks is written afterwards by a separate team that knows the full protocol, including the gap detector, but not the secret mapping. The sealed set includes partial-injection attacks tuned to stay inside the honest gap distribution, and attempts to smuggle instructions through numeric constants or code structure that survives canonicalization.
Baselines
The naive reference is the raw-view monitor feeding the adaptive credence protocol of Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats, run against the injection attacks of Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols. It is the published system, and the attack paper shows it can be driven to zero suspiciousness. Its clean-setting numbers (98.4% usefulness, 99.2% safety over 10,000 APPS tasks) are the target we aim to recover under attack. We found no reported result on Lacuna for any defence against monitor injection, so we will measure the attacked baseline ourselves.
A static upfront-auditing protocol with a fixed suspiciousness threshold serves as the non-Bayesian reference.
The strongest competitor is blinded-only monitoring, run at two strengths: comment and docstring stripping, and our full canonicalization scored through the same credence protocol. It removes the injection channel just as our method does, and it is the obvious thing a practitioner would try.
A hardened-prompt monitor, told explicitly to ignore instructions embedded in code, covers prompt-level defences.
To isolate our key ingredient we run two ablations. The first is two-view per-task gap thresholding without Bayesian accumulation. The second is a two-view model with a public rather than secret renaming map.
Every arm uses the same untrusted model, the same trusted monitor, identical task streams, the same audit budget, and the same number of tuning runs on the development split. All parameters are frozen before the sealed attacks are opened. Our method beats a competitor if it reaches higher safety at matched usefulness against the attacker's best response from the sealed set, where the attacker picks whichever strategy hurts each arm most.
Why not just…?
Why not just strip comments and strings and monitor the blinded code alone? This is the real threat to the proposal, and the experiments are built to settle it. If the blinded-only monitor reaches the same AUROC as raw monitoring against attackers who do not inject, then the second view adds nothing, and we would report that the contribution reduces to canonicalization plus credence. Our reason to expect a gap is that comments, names and strings carry intent. They help a weak monitor decide whether an odd special-case branch is justified. Since plan-only monitoring collapses to 0.52 when concrete context is withheld (Factor(U,T)), removing some context plausibly costs some detection. The experiment that decides it is the comparison on the non-injecting attack set.
Why not just harden the monitor's prompt to ignore embedded instructions? Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols shows that injections can be discovered against a specific monitor and transferred zero-shot. A prompt-level defence is just one more target for that search. We include it as a baseline, and expect the sealed adaptive attacks to defeat it.
Why not just have a trusted model rewrite the code before monitoring? The rewriter reads the injected text and can be hijacked in exactly the same way. Resampling-style mitigations that hand the attacker more attempts made things worse in the attack paper. A deterministic rewriter has no instruction-following surface, which is the whole point.
Proposed Experiments
The primary testbed is the APPS backdooring setting (Measuring Coding Challenge Competence With APPS), with 10,000-task streams that mirror Wen et al. The secondary testbed is the 342-task BigCodeBench subset used by Factor(U,T) (BigCodeBench, code at bigcodebench), where code monitoring reaches 0.96 AUROC and the unaudited attack success rate under trusted decomposition is 1.2%. BashBench tests whether the method carries over to a language where free text is harder to separate from logic.
We report per-task AUROC for each view and for the joint score, deployment safety and usefulness curves, and the attacker's best-response safety. We also report the empirical spread of honest raw-versus-blinded disagreement, which bounds how much a stealthy partial injection can buy.
The decisive checks are three. Against non-injecting attacks, two-view monitoring should match the raw monitor and beat blinded-only. Against injection attacks, it should match or exceed blinded-only, while the raw-view baseline collapses. Against the sealed adaptive set, safety at matched usefulness should stay close to the clean-setting numbers of Wen et al., and the secret-map ablation should show whether hiding the canonical form matters.
If the first check fails, the right recommendation is plain canonicalization, and we will say so.
References
- Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats
- Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols
- Factor(U,T): Controlling Untrusted AI by Monitoring their Plans
- Combating False Reports for Secure Networked Control in Smart Grid via Trustiness Evaluation
- The Beta Reputation System
- Improved Measurement of the<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:mi>π</mml:mi><mml:mo stretchy="false">→</mml:mo><mml:mi mathvariant="normal">e</mml:mi><mml:mi>ν</mml:mi></mml:mrow></mml:math>Branching Ratio
- Spectral Measurement of Electron Antineutrino Oscillation Amplitude and Frequency at Daya Bay
- New Measurement of the<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:msup><mml:mi>K</mml:mi><mml:mo>+</mml:mo></mml:msup><mml:mo>→</mml:mo><mml:msup><mml:mi>π</mml:mi><mml:mo>+</mml:mo></mml:msup><mml:mi>ν</mml:mi><mml:mover accent="true"><mml:mi>ν</mml:mi><mml:mo>¯</mml:mo></mml:mover></mml:math>Branching Ratio
- CellAgent: LLM-Driven Multi-Agent Framework for Natural Language-Based Single-Cell Analysis
- Self-Preference Bias in LLM-as-a-Judge
- Measuring Coding Challenge Competence With APPS
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions