Habituating to the Right Signal: Contract-Aware Gates for Reviewing Agent Code

Developers who repeatedly review pull requests written by AI coding agents grow more lenient over time. Approvals rise and comments shrink, and this does not happen with human-authored code reviewed by the same people. This proposal treats that drift as something other than a willpower failure. Following a normative model in which habituation is optimal filtering, the reviewer may be filtering the wrong evidence. Passing tests and tidy diffs say very little about the failure mode that actually breaks downstream code: quiet changes in how a function handles bad inputs and errors. We propose to build and test a review gate that supplies reviewers with evidence on that dimension. The gate is a separate, swappable stage in the review pipeline. It runs old and new versions of each changed function on boundary inputs, keeps a running estimate of each agent's contract risk, and raises scrutiny only when that estimate warrants it.

The bet is specific. A good gate should not stop habituation. It should make habituation track real risk, so reviewers relax when agents are in fact safe and snap back when contract drift appears. We test this in three ways. A retrospective analysis of real review logs tries to tell habituation apart from earned trust. An offline benchmark tests whether automated contract probes predict downstream failure. A controlled study with human reviewers compares the gate against simpler fixes at a matched interruption budget.

Core Claim

Habituation as optimal filtering models an agent that tracks a hidden danger level $x(t)$ from noisy observations. Its response is the posterior probability that danger exceeds a threshold $\psi$:

$$P(x(t) > \psi \mid y_{1:t}) = 1 - \Phi\!\left(\frac{\psi - \mu(t)}{\sigma(t)}\right),$$

where $\mu(t)$ and $\sigma(t)$ are the posterior mean and standard deviation after observations $y_{1:t}$, and $\Phi$ is the standard normal cumulative distribution. Repeated benign-looking observations shrink $\sigma(t)$. When $\mu(t)$ sits below $\psi$, the response then collapses, which is the model's account of habituation.

That paper evaluates the model only on synthetic stimulus sequences. Its premise is that the observations $y$ are informative about $x$, and code review breaks that premise. Is Agent Code Less Maintainable Than Human Code? shows two things. Downstream failures of agent-written code are driven by Input/Error Contract drift, meaning changes such as returning None instead of raising an exception, which raise failure odds 1.83x. Conventional signals like cyclomatic complexity and Halstead volume, which measure how branchy or verbose code looks, do not predict those failures.

A reviewer who updates on "tests pass, code looks clean" is therefore shrinking uncertainty along a channel that is nearly uninformative about the risk that matters. That is the situation in which optimal filtering produces confident blindness. The claim is that adding a contract-level observation channel restores the model's useful properties. When no drift is present, habituation stays and saves effort. When drift appears, the new evidence shifts $\mu(t)$ toward the threshold, and the model predicts dishabituation, meaning sensitivity returns immediately.

This differs from the closest work in clear ways. Habituation at the Gate measures the leniency drift but proposes no intervention. It also cannot separate habituation from rational trust in agents that improved. We address both. Addressing vital sign alarm fatigue using personalized alarm thresholds fights alarm fatigue by learning static, offline per-patient thresholds. Our gate instead updates online, and it changes what evidence the human sees rather than only when an alarm fires.

Research Program

The work has three components.

Diagnosing habituation versus earned trust. We need a ground-truth quality signal for agent PRs. We extend the AIDev review logs used by Habituation at the Gate with post-merge repository history, marking a merged PR as defective if a revert or a fix commit touches its lines within a fixed window.

The two explanations then make opposite predictions. Rational trust calibration says that among PRs a reviewer approves, the defect rate should stay flat as the reviewer gains experience, because leniency rises only as agent quality rises. Habituation says the defect rate among approved PRs should rise with reviewer experience. We also fit the filtering model per reviewer. The fitted model gives the approval trajectory that a fixed-threshold Bayesian reviewer would produce from the outcomes that reviewer could actually observe. The excess of real approvals over that trajectory is our habituation index.

A contract probe. For each changed function, the probe runs the pre-change and post-change versions on generated boundary inputs: empty collections, None, wrong types, out-of-range values, and inputs that previously raised exceptions. It then reports differences in exception types, return types, and guard behavior.

This is differential testing aimed at the specific failure that Is Agent Code Less Maintainable Than Human Code? identified. We validate it on that paper's CodeThread chains. CodeThread turns benchmark issues into two-step PR chains, where an agent first reimplements a function and a second agent then solves a dependent issue on top of it.

The gate. The gate keeps a filtering posterior over contract risk for each agent and repository. It updates from probe outputs and from downstream outcomes. Its exceedance probability sets how prominently contract differences are shown to the reviewer. The gate also tracks the reviewer's own scrutiny, using comment volume and active inspection time. It inserts a known-drift canary PR only when reviewer scrutiny has fallen faster than the gate's risk estimate has.

The gate is built as decoupled stages (probe, risk filter, presentation) passing a shared intermediate record. This follows the staged design of A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation, which splits evaluation into separate stages joined by a common schema. The modular design is here to serve the science. Each stage can be ablated or replaced by a baseline without touching the others, and that is what makes the comparisons below clean. The rest of the experimental scaffolding (fixed seeds, recorded commit hashes) follows the reproducibility practices in DeepDIVA.

Baselines

Unsupported review. The do-nothing reference is the reviewer with only the standard diff and test results. Its real-world behavior is documented: Habituation at the Gate reports a +14.5 percentage point rise in approval rate from the first to the tenth experience decile, a 22% drop in inline comments, and a 28% drop in comment words. In our controlled study this arm uses the same PR stream as every other arm. Any arm must flatten the decline in drift-catch rate across the session relative to this arm to count as an improvement.

Static complexity flags. This baseline flags PRs by cyclomatic complexity and Halstead volume. It is what most teams' tooling would offer. Is Agent Code Less Maintainable Than Human Code? reports these metrics as not significantly predictive. No reported number for their predictive power was found on Lacuna. We recompute their predictive performance on the same CodeThread instances as the probe. The probe beats this baseline if its area under the ROC curve for predicting downstream failure is significantly higher.

An LLM judge. A language-model judge reads each diff and writes a risk summary. It runs on the same PRs, with a prompt tuned on a held-out split using the same tuning budget we give the probe's input generator. The closest reported judge figure is 85.0% accuracy for the 0.6B OmniEval Verifier on a human-verified split, from A Composable Evaluation System. That figure measures answer grading, not code risk, so it only bounds expectations. No reported result was found on Lacuna for LLM judges detecting contract drift.

Always-on contract warnings. This is the strongest competitor and the key ablation. Reviewers see every probe output at full prominence, with no adaptive filter. If it matches the full gate, the habituation modeling adds nothing beyond the probe itself.

Fixed-rate canaries. Known-drift canary PRs are inserted at the same total count the adaptive gate uses, but at uniform random positions. This isolates whether the timing of interruptions matters.

The full gate counts as winning if, at an equal interruption budget, its drift-catch rate in the final third of the session exceeds both the always-on and fixed-canary arms, without an increase in false rejections of clean PRs.

Why not just…?

Why not just run the contract probe and auto-block, removing the human? For drift the probe can see, this may be the right answer, and we will report how much it catches alone. But drift is not always a bug. Some contract changes are intended, and only a human with project context can say which. Auto-blocking every contract difference recreates the false-alarm problem from intensive care monitoring, where Addressing vital sign alarm fatigue cites estimates that 64% to 99% of alarms are clinically insignificant. The CodeThread analysis reports how many probe-flagged differences are benign. That figure settles how much human judgment is still needed.

Why not just show the warnings all the time? Constant warnings are another alarm stream. The filtering account predicts that reviewers will habituate to an alarm whose firing is uninformative. This is a real threat. The always-on arm tests it directly, and if it matches the gate, we will say the probe is the contribution and the adaptive filter is not.

Why not just audit at a fixed random rate, or rotate reviewers? Random audits are cheap and well understood, and they may work about as well. The matched-budget fixed-canary arm is designed to show that, so a null result here is informative rather than fatal. Reviewer rotation addresses the time a reviewer has spent with one agent but not the uninformative channel. We include a rotation condition in the retrospective analysis, by comparing reviewers who split reviews across several agents. We do not include it in the lab study.

Why not just call it burnout? Habituation at the Gate already rules out calendar-time fatigue. Approval of human-authored PRs from the same reviewers in the same repositories stayed flat or declined over the same period.

Proposed Experiments

Retrospective test on AIDev. The decisive check is the defect rate among approved agent PRs as a function of reviewer experience decile. The comparison is within reviewer, controlling for agent and repository, with human-authored PRs as a difference-in-differences control. A rising rate supports habituation. A flat rate supports earned trust and would weaken the motivation for any intervention. We will publish either outcome.

Offline test on CodeThread. CodeThread's 1,377 instances span SWE-Bench Verified, Multilingual, and Pro plus FeatBench. On these we measure precision and recall of the probe against the paper's drift annotations. We also measure how well the probe predicts downstream failure, against the reported effects of 1.83x failure odds under drift and a resolve-rate drop of up to 13.1% (GLM 4.7, SWE-Bench Pro). No reported result was found on Lacuna for automated drift detection, so the probe's numbers will be the first.

Controlled human study. Developers review a long stream of agent patches taken from CodeThread's first step, with naturally occurring drift at a fixed, low rate. We use a between-subjects design across the five arms and pre-register a power analysis. The primary metric is the slope of drift-catch rate over position in the stream. Secondary metrics are false rejections, active time per PR, and comment volume.

A second block contains no drift at all. There, the filtering model predicts that the full gate should let review time fall, preserving the efficiency of habituation. The always-on arm should not show that fall. When drift is then reintroduced, the gate arm should show sharp recovery of catch rate. Observing both effects would confirm the claim that the goal is habituation tied to the right signal, not the elimination of habituation.