In the rapidly evolving world of AI, we are moving from chatbots that talk to Computer-Using Agents (CUAs) that do. These agents navigate websites, manage files on desktops, and interact with mobile apps. But there’s a catch: how do we know if they actually finished the job?
A trajectory might look successful in a screenshot, but the file might not have actually saved, or the agent might have fulfilled only half of a complex instruction. Current verification methods—like asking a Vision-Language Model (VLM) to judge a few screenshots—often miss these subtle failures.
Enter VeriCUA, a new framework from researchers at Google and The University of Hong Kong that treats verification not as a single snapshot judgment, but as an active exploration task.
TL;DR
VeriCUA is a platform-agnostic framework that verifies computer-using agents by treating verification as an agentic exploration process. By dynamically gathering evidence through both visual (GUI) and programmatic (code-based) checks, it significantly reduces false positives and outperforms static judges across web, desktop, and mobile environments.
What Problem Does This Paper Address?
Verifying if an agent completed a task is surprisingly hard. Traditional "VLM judges" suffer from a partial view problem: they only see the screenshots provided to them. If the decisive evidence (like a specific database entry or a file deep in a folder) isn't in those frames, the judge guesses.
On the other hand, programmatic checkers (scripts that check for a file's existence) are often too rigid. They might look for a file at a specific path, but if the agent saved it elsewhere successfully, the script fails. Furthermore, building custom verifiers for every single app and platform is an engineering nightmare. We need a verifier that is as flexible and exploratory as the agents it's supposed to judge.
Methods: Verification as Exploration
VeriCUA flips the script by turning the verifier into an agent itself. Instead of a single pass, it follows a structured, four-stage "agentic" workflow:
- Context Inspection: Before judging, the verifier "scouts" the environment. It looks at available files, database schemas, and UI trees to understand the specific context of the application.
- Rubric Synthesis: Using the original instruction and the scouted context, it generates a fine-grained rubric of "hard" (mandatory) and "soft" (optional/hygiene) requirements.
- Active Exploration: The verifier then selects tools from a Hybrid Action Space to gather evidence. It might use GUI tools (like zooming in on a specific UI element or performing OCR) or Code tools (like querying an SQL database or checking file metadata).
- Rule-Based Arbitration: Finally, it combines these findings. Crucially, it distinguishes between asserted evidence (what the agent claimed it did) and attested evidence (what the environment actually proves happened).
This hybrid approach allows the verifier to see what a recording might have missed. For example, a screenshot might show a "Save Successful" message (asserted), but a code-based check can confirm the file's timestamp actually updated (attested).
Datasets
The researchers evaluated VeriCUA across a wide range of benchmarks to prove its platform-agnostic nature:
- Online (Live Environments): AndroidWorld (mobile), OSWorld (desktop), and ScienceBoard (scientific desktop apps).
- Offline (Recorded Trajectories): WebArena (web) and OSReward (a cross-platform human-labeled dataset).
Key Insights and Ideas
- Scouting Prevents Hallucinations: By inspecting the environment before creating a rubric, VeriCUA avoids assuming file paths or UI behaviors that don't exist in that specific app.
- The Power of "Abstention": Unlike standard VLM judges that must give a "Success" or "Fail" answer, VeriCUA can abstain if the evidence is inconclusive. This makes its "Decided Accuracy" much higher, which is vital for high-stakes data curation.
- Hybrid Evidence is Non-Redundant: The study found that adding code-based state checks to GUI inspection increased the decision rate and recall without increasing false positives. Visuals corroborate the "what," while code confirms the "is it really there?"
Key Results
VeriCUA consistently outperformed both standard VLM judges and automatically generated scripts.
- Higher Accuracy: In the AndroidWorld suite, VeriCUA achieved 90.5% accuracy, compared to 78.4% for a standard VLM judge.
- Lower False Positives: In ScienceBoard, VeriCUA's false-positive rate was just 5.4%, whereas the VLM judge flagged nearly 25% of failures as successes.
- Catching Liars: VeriCUA is particularly good at rejecting "false completion claims"—cases where the agent says "I'm done!" but hasn't actually finished. It rejected ~80% of these reports across online environments.
Limitations and Future Work
While powerful, VeriCUA is not a silver bullet.
- Cost: Because it performs multiple steps and tool calls, it is significantly more expensive (in terms of API tokens and time) than a single-pass judge. It is best suited for offline evaluation or reinforcement learning rather than real-time interactive use.
- Read-Only Constraint: To ensure safety, VeriCUA never "writes" to the environment it verifies. While this prevents the verifier from breaking things, it also means it can't perform "destructive" tests (like trying to delete a file the agent was supposed to protect).
Future work may involve optimizing the cost of these agentic passes and expanding the toolset to even more complex enterprise software environments. For now, VeriCUA represents a major step toward building agents we can actually trust to get the job done.
Summary updated