Computer-use agents are AI models that operate desktop software by reading screenshots and clicking and typing. They are graded today the way a short clinical trial grades a drug. Each task runs for a minute or two, someone checks the final screen state, and the episode ends. In medicine, short follow-up is known to mislead. An early improvement in a surrogate marker, such as heart remodeling at six months or cognitive scores at six months, often fails to predict the outcome that actually matters later. We propose to measure whether single-episode agent benchmarks are the same kind of misleading surrogate. The test is to run agents through a simulated multi-week laboratory study in which each week's work builds on the last.
The testbed is Western blot quantification. A Western blot is a lab assay that reports protein amounts as dark bands on a membrane image. To quantify one, a scientist measures band intensity in image software, divides by a loading-control band, and appends fold-changes to a running spreadsheet. This workflow suits our question for three reasons. Its errors are numerical and silent: a wrong normalization in week 2 corrupts every later figure without breaking anything visible. Its ground truth can be controlled exactly, because synthetic blots can be rendered with known band intensities. It also has a clear "hard endpoint": whether the study's final scientific conclusion is correct.
Core Claim
Our hypothesis is that per-session success rates on short tasks systematically overestimate how reliable an agent is across a multi-session deployment. The error is not explained by naive compounding of independent failures. It comes from correlated, state-dependent failures: corrupted spreadsheets, drifting file conventions, and changed dialogs that one bad session plants and later sessions inherit. If this holds, single-episode benchmarks are surrogate endpoints in the clinical sense, and agent evaluation needs follow-up designs. If it fails, meaning observed long-run reliability matches what independent per-session rates predict, then single-episode benchmarks are adequate and the community can simply multiply. Both outcomes are informative.
The precise test is simple. Let $p_t$ be an agent's measured success rate on session $t$ when that session starts from a clean, correct state. Let $S(T)$ be the observed fraction of full $T$-session runs whose final study conclusion is correct. If failures were independent and always caught, a run would succeed exactly when every session succeeds, giving
$$\hat S_{\text{indep}}(T) = \prod_{t=1}^{T} p_t .$$
Our claim is that $S(T)$ departs from $\hat S_{\text{indep}}(T)$ in a direction that the per-session numbers cannot reveal. The ratio between the two, tracked as $T$ grows, is the quantity we report.
Where the Pieces Come From
The benchmark design borrows from three bodies of work, each in a different role.
Existing agent benchmarks motivate the gap and supply the evaluation style. OSWorld runs agents in real Ubuntu and Windows virtual machines and checks final state by executing scripts. Its tasks have a median human completion time of about 112 seconds, and the authors report memory drift and action loops as trajectories lengthen. VisualWebArena does the same for websites. ScienceBoard adds scientific software such as ChimeraX, with probes inside the application backend. None of the three evaluates more than one session, and none includes bio-image or gel analysis. We reuse their execution-based checking but extend it across sessions with persistent state.
Clinical work on short follow-up supplies the methodological lens. The CMR dyssynchrony study found an imaging index that strongly predicted six-month remodeling (odds ratio 5.67) but could say nothing about mortality. The meta-analysis by Hu et al. shows the same pattern for cardiac imaging proxies. ADL+ reports cognitive gains at 6 months that it could not confirm over a 3-month follow-up. We take from these papers the distinction between a surrogate and a hard endpoint, and the practice of reporting both.
Western blot forensics supplies the ground-truth machinery and a design constraint. Forensic Analysis of Synthetically Generated Western Blot Images shows that diffusion models and GANs generate photorealistic blots. A one-class detector still separates them from real blots (AUC 0.999 on DDPM images), so we must check that our synthetic blots are not trivially artificial. FBI_GEL argues that integrity checks should be deterministic: identical outputs for identical inputs. We adopt that principle for our auditor condition, described below. The study template follows Jiao Tai Wan, an 8-week rat study that quantified AMPK, pAMPK, ACC, pACC, and FAS. In that template phospho-proteins are normalized to their total protein, which creates exactly the multi-step normalization chain where silent errors propagate.
Research Program
We will build a virtual-machine environment containing Fiji/ImageJ and LibreOffice Calc. A run simulates an 8-to-12-week time-course study. Each simulated week, new blot images for the five target proteins and a loading control arrive in a shared folder. The agent must do four things: measure band intensities, normalize phospho-to-total and to the loading control, append the values to the cumulative spreadsheet, and regenerate a fold-change chart.
Blots come from two sources. The first is parametric renders with exactly known band volumes, passed through a generative model for realism. These are realism-checked with the one-class detector so that they are not trivially separable from real blots. The second is a smaller set of real blots with expert densitometry as reference. Each run has a planted ground-truth effect, such as a treatment that raises pAMPK/AMPK and lowers FAS. This defines the hard endpoint: whether the final spreadsheet supports the correct direction and rough size of each effect.
Between sessions we introduce realistic drift: a renamed file convention, a software update that moves a dialog, a collaborator's edit to the spreadsheet, and an occasional duplicated lane of the kind FBI_GEL targets. Some of these require the agent to read the workspace rather than follow its earlier routine.
Every session is scored twice. The session score asks whether this week's numbers are correct given correct inputs. The endpoint score asks whether the study conclusion is correct at week $T$. We also record time to the first silent corruption, meaning an error that survives into later sessions, and analyze it as a survival curve.
Baselines
The naive reference point is a vision model that skips the software. It reads band intensities directly from a screenshot of the blot and writes estimates into the spreadsheet. This shows whether software operation adds anything over visual eyeballing. Any agent that cannot beat it on endpoint correctness has failed outright.
The ceiling reference is a deterministic Fiji macro plus a Python script that computes every quantity from raw images. Agents cannot match it on accuracy. It defines the achievable endpoint and serves as the source of truth for scoring.
The strongest existing agent is the modular planner-actor design from ScienceBoard. A general model (GPT-4o) plans, and a dedicated GUI model (GUI-Actor-7B) clicks. ScienceBoard reports 20.44% overall success for this design, the best on that benchmark, against 60.27% for humans. We also run Claude-3.7-Sonnet with screenshot plus accessibility tree, which ScienceBoard reports at 15.79%, the best standalone configuration there. We run the open ScaleCUA model as well (ScaleCUA; it reports 60.6% on the OSWorld-G grounding benchmark). There is no reported result on Lacuna for any agent on Western blot or gel-quantification GUI tasks, so all of our numbers on this environment are new.
For fairness, every agent gets the same images, the same virtual-machine snapshot, the same per-session step budget, and the same prompt-tuning budget. That budget is spent on a held-out development study, never on test runs. We use at least 20 seeded runs per agent so that endpoint rates have usable confidence intervals.
The key ablation isolates compounding. Each agent runs the identical sequence of sessions twice. In the reset condition, the workspace is restored to ground truth before every session, which yields $p_t$. In the persistent condition, the agent inherits whatever it left, which yields $S(T)$. Only state carry-over differs between the two.
A second ablation varies the agent's view of the screen: screenshot only, accessibility tree only, and hybrid. It tests whether the hybrid advantage ScienceBoard found on single episodes survives over many sessions. Java interfaces like Fiji's may expose sparse accessibility trees, so the answer is not obvious.
A third condition adds a deterministic auditor. After each session, a script recomputes values from the raw images and flags disagreements to the agent. This measures how much long-run reliability returns when correctness checking is moved out of the stochastic model, in the spirit of FBI_GEL.
Beating the baselines means two things. First, the persistent-condition analysis must show a gap between $S(T)$ and $\hat S_{\text{indep}}(T)$ whose confidence interval excludes zero. Second, single-session rank and endpoint rank must disagree for at least one pair of agents. Either result would show that single-episode scores mislead.
Why not just…?
Why not just run the densitometry as a headless script? For doing the science, a script is better, and we include one as the ceiling. The proposal is not that agents should quantify blots. It uses blot quantification as a controllable instrument for measuring agent reliability over time. Many labs do work through graphical tools, and ScienceBoard reports that agents do markedly better where command-line shortcuts exist. We therefore also run a condition in which agents may write and run macros. If that condition closes the long-run gap entirely, that is a substantive finding: tool choice, not perception, drives durability.
Why not just compute pass^k, or multiply single-episode success rates? This is the real threat. If failures are independent and errors are detected immediately, the product of reset-condition rates fully predicts long-run success, and our benchmark collapses into existing ones. The reset-versus-persistent ablation is designed to settle exactly this question, and we will report it even if the gap is null. The clinical record suggests it will not be null. ADL+ and the CMR study show early measurements failing to carry forward. More directly, silent numerical errors persist rather than vanishing, which violates the independence assumption.
Why not just give the agent longer memory or a bigger context window? Memory is a treatment, not an evaluation. Without a longitudinal benchmark nobody can tell whether it helps. We include a memory-augmented variant of the strongest agent as one arm. The prediction to test is that memory reduces failures that come from forgetting conventions but not failures that come from inherited corrupted state. The deterministic-auditor arm should target the second kind.
Why not just reuse OSWorld with repeated tasks? OSWorld's tasks are independent and its checks only see final state. Repeating them creates no shared state, so nothing compounds. OSWorld also has no notion of a downstream conclusion that can be right or wrong while each screen looks fine. The cumulative Western blot study supplies both of these.
Decisive Checks
The first check is the reset-versus-persistent gap at $T = 8$ and $T = 12$, reported per agent with bootstrap intervals. The second is the rank correlation between single-session success and endpoint correctness across agents and view conditions. A low correlation is the clearest evidence that single-episode benchmarks act as a misleading surrogate. The third is the survival curve of time to first silent corruption, compared between the plain agents and the auditor arm.
A sanity check guards the whole design. The synthetic blots must not be trivially detectable as synthetic by the co-occurrence detector, and real-blot runs must reproduce the qualitative pattern seen on synthetic runs. If real and synthetic blots diverge, the conclusions are limited to synthetic data and we will say so.
References
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
- VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
- ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows
- A prospective evaluation of cardiovascular magnetic resonance measures of dyssynchrony in the prediction of response to cardiac resynchronization therapy
- Comparative efficacy of image-guided techniques in cardiac resynchronization therapy: a meta-analysis
- ADL+: A Digital Toolkit for Multidomain Cognitive, Physical, and Nutritional Interventions to Prevent Cognitive Decline in Community-Dwelling Older Adults
- Forensic Analysis of Synthetically Generated Western Blot Images
- Forgery Blind Inspection for Detecting Manipulations of Gel Electrophoresis Images
- Jiao Tai Wan Attenuates Hepatic Lipid Accumulation in Type 2 Diabetes Mellitus
- ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data