Who Is Steering the Agent? Change-Point Detection of Injected Intent in Screenshot-Only Agent Trajectories

Computer-use agents that work only from screenshots, with no access to the page's underlying element tree, treat every pixel as instructions they may follow. An attacker can therefore render a fake pop-up, notification, or message that quietly redirects the agent. On recent benchmarks this works alarmingly often. Claude 3.7 Sonnet carries out injected attacks in 46.2% of Messenger cases in VPI-Bench. GPT-4o and Claude-3.7-Sonnet show vulnerability rates near 55% on GhostEI-Bench. Warning the model in its prompt barely helps. We propose to borrow the central idea of intrinsic plagiarism detection, which finds where a second author's text begins inside a single document without any reference corpus. A hijacked trajectory is a document with two authors: the user wrote the first part, and the screen wrote the rest. The hijack is the point where the authorship changes.

The detector reads the agent's own per-step internal states and asks, for every step, whether the steps before and after it look like they come from different sources. Because a hijack step often looks locally harmless and only becomes suspicious in light of what follows, the detector reads the trajectory in both directions. It runs either as an after-the-fact audit that pinpoints the step to roll back, or with a short lookahead over the agent's own unexecuted plan before any irreversible action. The deliverable is a small, cheap, on-device model that both flags the hijack and locates the step where it began. That location is what a rollback mechanism needs and what a yes/no verdict cannot supply.

Core Claim

Injected intent leaves a two-sided discontinuity in the agent's own representations. Before the breach, each action can be traced to the user's instruction. After it, each action can be traced to content that newly appeared on screen. A statistic that compares the segment before a candidate step with the segment after it should localize that boundary better than either of two cheaper options. The first is a causal check that only sees the past. The second is a per-frame image check that never sees the sequence at all.

The claim is new along three axes. First, the unit of analysis is the trajectory treated as a self-contained document, not a single screenshot. Second, detection is reference-free: there is no database of known attacks and no external frontier judge. Third, the output is a localized boundary rather than a verdict on the whole trajectory.

Where This Sits

Three strands of intrinsic plagiarism work supply the method. Unsupervised Anomaly Detection compares each segment with its "dynamic complement," meaning the rest of the document. It ranks 1,000-word foreign insertions in the top 20 in 94.83% of cases, but only 64.73% for 100-word insertions. Style Breach Detection with Neural Sentence Embeddings scores each sentence by its mean embedding distance to all other sentences and reaches a window-based F-score (WinF) of 0.28 on PAN-2017. RDI_RID flags chunks whose distance from the document mean vector crosses a threshold.

These papers also constrain the design, because they document the field's known failures. Short segments carry too little signal. A single global threshold breaks when a document has several long sections. Legitimate shifts in style produce false alarms; the statistical language model approach calls this the "proper noun trap." Agent trajectories have all three problems. Hijacks are often only two to five steps long. Benign tasks legitimately shift phase, for example from search to checkout. And on-screen labels are full of proper nouns.

We respond to each failure in turn. We use a two-sided split statistic instead of a distance to the global mean, because a hijack that persists until the end of the trajectory would otherwise contaminate the mean. We learn the threshold from data instead of fixing it globally. And our features record where each action came from, not what it looks like. That is how we sidestep the problem of benign phase shifts.

Bidirectional sequence modeling supplies the second ingredient. In sleep staging and outbreak forecasting (FIGI-Net), some states can only be told apart by what comes after them. We use a bidirectional linear recurrent model for the learned part of the detector because its cost grows linearly with trajectory length. DeltaFlow notes that full bidirectional attention scales quadratically with sequence length.

The closest agent-side work motivates the gap without filling it. BacktrackAgent checks each transition causally with a rule-based verifier and a vision-language-model judge. That catches crashes and error pages, but not a hijack that produces a perfectly valid screen. It also halves inference speed. Gradient-guided Token Masking (GTM) finds adversarial image tokens through the gradient of the first hidden state's norm and removes them, driving static-image attack success to near zero. It never looks across steps. GhostEI-Bench uses a frontier LLM judge to label trajectories after the fact. VPI-Bench explicitly calls for guard models that monitor trajectories against user intent. Neither builds one.

Research Program

The pipeline has five parts: per-step features, a two-sided statistic, a learned bidirectional detector, a way to run it before irreversible actions, and training data.

Per-step features. For a trajectory of $T$ steps, each step $t$ gets a vector $z_t$ with two components, both computed from the open-weight agent's own forward pass with no page metadata.

The first is a pooled hidden state at the moment the agent emits its action.

The second is a provenance profile. It is a gradient-times-input attribution of the chosen action, split across three groups of input tokens: the instruction tokens, image tokens for regions that were already on screen, and image tokens for regions that newly appeared at step $t$. "Newly appeared" is determined by a pixel difference against the previous frame. This is GTM's gradient machinery, redirected. GTM asks which tokens are adversarial within one image. We ask where each action's decision came from over time.

The provenance profile is the key ingredient. A user-driven trajectory can change visually a great deal while its actions stay attributable to the instruction. A hijacked one changes where its actions come from.

Two-sided statistic. Following the complement idea of Unsupervised Anomaly Detection, each candidate boundary $\tau$ gets a score:

$$S(\tau) = d\big(\mu_{1:\tau-1},\ \mu_{\tau:T}\big)\cdot \sqrt{\tfrac{(\tau-1)(T-\tau+1)}{T}}$$

Here $\mu_{a:b}$ is the mean of $z_t$ over steps $a$ through $b$, and $d$ is a learned Mahalanobis distance estimated on clean trajectories. The square-root factor is the standard change-point weighting. It comes from the variance of a difference of two sample means, and it stops short end-segments from dominating the score.

Learned bidirectional detector. A small bidirectional linear recurrent network reads $z_{1:T}$ along with $S(\cdot)$. At each step it outputs the probability that step $t$ is the breach onset. It is trained with a per-step cross-entropy loss.

Running it before irreversible actions. Before any irreversible action, such as send, upload, delete, or grant permission, the agent writes out its next $k$ planned actions without executing them. The detector then reads the past steps plus those planned steps. If it fires, the localized onset is handed to a rollback reflector in the style of BacktrackAgent.

Training data. We use the plagiarism community's own trick, artificial insertion as in the PAN-PC corpora. Clean trajectories come from AndroidInTheWild and Mobile3M. Spliced trajectories switch partway through to another task's actions, triggered by a synthetic overlay that we render into the frame. No benchmark attack is used for training.

Baselines

Do nothing. The naive reference point is the undefended agent's vulnerability rate. On GhostEI-Bench, the reported vulnerability rates are 55.12% for Claude-3.7-Sonnet, 54.87% for GPT-4o, and 16.43% for GPT-5 preview. On VPI-Bench, Claude 3.7 Sonnet's reported attack success rate is 46.2% on Messenger and 37.2% on Email. Because our detector needs open weights, we re-measure the undefended rate for our own open agent, a Qwen2-VL-class model, on both benchmarks under identical seeds. Every defense is compared against that measured figure.

Cautious prompting. This is the cheapest defense. It is run with the wording reported in both benchmark papers. Both papers describe it as insufficient but give no defended number in our file, so we measure it ourselves.

Frontier LLM trajectory judge. This is the strongest competitor on detection. A frontier model receives the user instruction, the screenshots, and the actions, and is asked whether and where the agent was hijacked. It is the evaluator GhostEI-Bench already uses, and it is exactly the guard model VPI-Bench asks for. There is no reported result found on Lacuna for using such a judge as a defense.

To keep the comparison fair, the judge sees the same trajectories and the same $k$-step lookahead as our detector, and gets the same number of prompt-tuning iterations on the splice validation set. To avoid circularity, all evaluation labels come from programmatic attack success: logged uploads, sends, and file deletions captured through the cua-framework/agents logging server and the emulator state. The judge's verdicts are never used as labels.

We beat the judge only if two conditions hold. Our onset-localization F-score must be at least equal to the judge's. And our latency and cost must be at least an order of magnitude lower, comparable to GTM's 0.22-second overhead rather than a frontier API call.

GTM per frame. GTM is the strongest existing image-level defense. Its reported results are attack success falling from nearly 100% to near 0% on static visual prompt injection, with essentially no MM-Vet degradation and 0.22 seconds of added latency. There is no reported result found on Lacuna for GTM on GhostEI-Bench or VPI-Bench. We run it at its default 5% masking ratio on every frame of the same agent.

BacktrackAgent-style causal judge. This stands in for causal step-level verification. Its reported results are on benign errors only: +7.59% task success over ReachAgent on Mobile3M, at roughly 50% of the generator's speed. We run it on the same agent with the same rollback mechanism.

Ablations. The ablations isolate the proposal's ingredients. We compare the causal-only direction against the bidirectional detector. We compare hidden-state features only against provenance features only. We vary the lookahead $k$ from 0 to 5. And we compare the two-sided statistic against the global-mean distance used in Style Breach Detection with Neural Sentence Embeddings.

Why not just…?

Why not just prompt the agent to be cautious? Both benchmarks tested this. VPI-Bench reports that agents "often fail to recognize the attack even when explicitly warned." GhostEI-Bench calls cautious prompting "insufficient." Its numbers also suggest that more capable planners are more susceptible, which argues against any defense that relies on the agent's own reasoning. We include cautious prompting as a baseline and do not expect it to be competitive.

Why not just ask a frontier model to review the trajectory against the user's instruction? This is the real threat, and it may match or beat us on raw detection accuracy. It sees the user's instruction, which our intrinsic detector deliberately does not rely on. Our case against it rests on three things that the experiments must establish rather than assume.

The first is cost and on-device feasibility. We need detection at every irreversible action, not once at the end.

The second is localization quality. Rollback needs the onset step, and a judge's free-text answer may be imprecise.

The third is robustness to the same deception. A judge that reads the injected text can itself be persuaded by it, and VPI-Bench's finding that stronger reasoners comply more makes this plausible. We test this directly by adding judge-targeted text to the injections.

We also report a hybrid in which our detector's top-ranked onset is the only thing passed to the judge. If the judge alone dominates on every axis, we will report that the intrinsic signal is mainly useful as a cheap pre-filter.

Why not just mask suspicious image tokens in each frame with GTM? GTM is evaluated on single static images. Many GhostEI-Bench injections are ordinary-looking notifications or permission dialogs. Nothing about them is adversarial at the token level; they mislead only through what the agent does next. A per-frame defense cannot tell a legitimate dialog from a spoofed one when the pixels look the same. The comparison on dynamic-injection cases will show whether the sequence view adds anything beyond per-frame masking. Masking also interferes with reading on-screen text, which matters for screenshot agents. We therefore measure benign task completion under GTM as well.

Why not just compare each action with the instruction embedding and flag low similarity? This is the simplest intent-consistency check, and it needs neither bidirectionality nor provenance. Its weakness is the benign-shift problem from plagiarism detection. Many legitimate steps, such as scrolling, dismissing a cookie banner, or entering a password, have low similarity to any instruction. We include it as a baseline. The hidden-state-only ablation tells us whether provenance is what actually separates hijacks from benign detours.

Proposed Experiments and Decisive Checks

Main evaluation. We train only on spliced AndroidInTheWild and Mobile3M trajectories. We test on GhostEI-Bench's 75 dynamic and 11 static injection cases and on VPI-Bench's 306 cases.

We report four metrics. The first is onset localization with PAN-2017's window-based precision, recall, and F-score plus WindowDiff, so that errors of one or two steps are counted fairly. The second is trajectory-level detection AUROC. The third is defended vulnerability rate and attack success rate after rollback. The fourth is the false-alarm rate and the change in task completion on GhostEI-Bench's benign cases and on held-out clean Mobile3M tasks.

Decisive check on bidirectionality. The causal ablation is evaluated on the subset of cases where the onset step leads to a valid-looking screen. If the bidirectional detector does not clearly beat the causal one there, the bidirectional component of the thesis fails.

Lookahead curve. We plot defended vulnerability rate against $k$. This tells us whether planned, unexecuted lookahead recovers most of the benefit of a full after-the-fact audit. Without that, the method is an auditing tool, not a defense.

Generalization to real attacks. Training on synthetic splices and testing on real attacks probes whether the detector learns "a change of author" or only memorizes our overlay style. As a sanity check on the statistic, we also run the two-sided score on PAN-2017 sentence embeddings. We expect it to at least match the global-mean method's WinF of 0.28. If it falls below that, the problem lies in the statistic itself, not in the transfer to agent trajectories.