CLIFT: Conformal Self-Verification for Web Agent Training and Test-Time Scaling

Yifan Zhang, Yutong Dai, Viraj Prabhu, Zhiyuan Hu, Ran Xu, Zeyuan ChenarXiv 2026 · arXiv

TL;DR

CLIFT (Conformal Self-Verification) is a new framework that turns expensive LLM judge feedback into a reusable bank of "certified" natural language questions. This bank provides dense, interpretable rewards during reinforcement learning and enables reliable, judge-free trajectory selection at test time, significantly boosting the performance of open-source web agents.

What Problem Does This Paper Address?

Training web agents—AI models that can navigate browsers to complete tasks—is notoriously difficult. Standard Reinforcement Learning (RL) typically relies on a binary "success/failure" signal at the end of a long sequence of actions. This sparse feedback makes it nearly impossible for a model to perform "credit assignment"—knowing exactly which click or scroll led to success or caused a failure.

While frontier LLMs (like GPT-4o or Claude 3.5) can act as "judges" to provide richer, step-by-step feedback, they are too expensive to call thousands of times during training and are often unavailable or too slow at deployment (test time). CLIFT addresses this by creating a middle ground: a compact, reliable, and model-agnostic "verifier" that the agent carries with it.

Methods

CLIFT introduces a two-stage process centered around Conformal Self-Verification:

  1. Training with a Certified Bank: During RL training (using the GRPO algorithm), the agent is asked to answer a series of natural language YES/NO questions about its own actions (e.g., "Did I reach the checkout page?"). A Compositional Conformal Certifier (CCC) compares these answers to a high-quality (but expensive) training-time judge. Only questions that consistently correlate with the judge's success criteria are "certified" and given "trust weights." These weights are used to create a dense, per-step reward that helps the agent learn much faster than binary signals alone.
  2. Test-Time Scaling (CTS): At deployment, the external judge is removed. The agent generates a "greedy" rollout and one or more "retries." It then uses the frozen bank of certified questions to evaluate these trajectories. A Conformal Trajectory Selection (CTS) mechanism uses a conservative majority-vote rule to decide whether to stick with the initial attempt or swap to a retry, all without needing an external LLM judge.
Figure 2: CLIFT overview. Training: rollouts are scored by a comparative judge and by a self- verifier that queries a polarity-tagged question bank. The CCC certifies question weights by URL tier, yielding a signed verifier score that is asymmetrically added to the judge reward and then converted into URL-stratified token advantages. Test-time scaling: the trained agent emits a greedy rollout and diverse retries; the frozen certified bank produces per-URL evidence for each pairwise comparison, and a conservative repeated-vote rule decides whether to keep the incumbent or swap.

The system is "URL-aware," meaning it tracks the reliability of verification questions at different levels: global, per-host (e.g., google.com), and per-URL-path (e.g., /search).

Datasets

The authors evaluated CLIFT across three distinct settings to test training, transferability, and zero-shot performance:

  • WebArena Infinity (WAI): A primary training benchmark featuring SaaS-style apps (like Gmail and GitLab) with programmatic verifiers.
  • VisualWebArena (VWA): A controlled transfer setting using Dockerized environments (Reddit, Shopping) to see if a bank trained on an open model (Gemma) helps a closed model (GPT-5.5).
  • Online Mind2Web (OM2W): A "hard mode" benchmark consisting of 300 live-web tasks across 30 domains, used to test how well the question bank transfers to entirely new websites without any policy fine-tuning.

Key Insights and Ideas

  • Interpretability over Black Boxes: Unlike traditional Reward Models that output a single number, CLIFT produces a "sheet" of evidence. You can see exactly which verification questions passed or failed, making the agent's reasoning auditable.
  • The "Additive" Reward: The training reward is designed to be asymmetric—it can add evidence to the judge's baseline but never subtract from it. This prevents the "reward collapse" often seen when blending different supervision signals.
  • Knowledge Transfer: A key "aha moment" is that the verification bank is model-agnostic. A bank built while training a smaller open-source model (Gemma-4) can be used to improve a much stronger closed-source model (GPT-5.5) at test time.
  • Conservative Selection: The CTS mechanism is intentionally biased toward the "greedy" rollout. It only swaps to a retry if there is repeated, certified evidence that the retry is better, acting as a safety valve against verifier hallucinations.

Key Results

CLIFT achieved state-of-the-art (SOTA) results across all tested benchmarks:

  • WAI: CLIFT-trained Gemma-4 reached a 74.6% success rate, outperforming Gemini 3 Flash and vanilla GRPO. The training reward alone provided a +7.9 percentage point (pp) gain over the baseline.
  • VisualWebArena: By transferring the Gemma-trained bank to GPT-5.5, the authors reached a SOTA weighted average of 53.7%.
  • OM2W (Live Web): This was the most impressive result. Without any training on the benchmark, simply translating and recalibrating the question bank improved GPT-5.5's success rate from 49.7% to 61.0% (+11.3pp).
Figure 4: WAI progression and gains. Operating points and deltas.

Limitations and Future Work

  • Judge Dependency: The initial quality of the bank depends on the training-time judge. If the judge is systematically biased, the certifier can only detect disagreements based on observed data, not fundamental flaws in the judge's logic.
  • Natural Language Bottleneck: While natural language is interpretable, it can be ambiguous. The authors suggest that a natural next step is replacing these questions with executable browser-state assertions (code) written by the agent itself.
  • Domain Specificity: While the bank transfers well, it isn't "universal." New web domains with radically different UI patterns might require a new round of question "evolution" and recalibration.

Summary updated

Abstract

Open-source web agents are now strong enough to execute realistic browser tasks, but training them with reinforcement learning still depends on weak supervision: binary task success is too sparse for credit assignment, while frontier-language-model judges are too expensive to call at every step and cannot be assumed available at deployment. We introduce CLIFT, a training and test-time scaling method built around conformal self-verification. During training, the agent answers natural-language verification questions about its own rollouts; a Compositional Conformal Certifier keeps only question signals whose URL-conditional evidence agrees with a training-time judge, assigns signed trust weights through polarity-aware lift, and blends the resulting verifier score into per-step rewards in a way that never subtracts from the judge baseline. At test time, the same certified bank is frozen and reused as structured evidence for Conformal Trajectory Selection (CTS): the agent samples a greedy rollout and one or more diverse retries, the self-verifier summarises each URL trace, and a conservative majority-vote rule chooses whether to swap away from the current incumbent without calling any external judge. This single mechanism supports three settings. On WebArena Infinity, CLIFT achieves state-of-the-art performance among open-source web agents. On VisualWebArena, a bank trained with the open model transfers to GPT-5.5 at test time and reaches state-of-the-art performance under the canonical harness. On Online Mind2Web, without training an agent on the benchmark, translating the certified question bank improves a live-web agent in zero-shot evaluation. Together these results position conformal self-verification as a way to turn costly judge feedback into a reusable training signal and a judge-free test-time scaling signal.
BibTeX
@misc{zhang2026clift,
  author = {Yifan Zhang and Yutong Dai and Viraj Prabhu and Zhiyuan Hu and Ran Xu and Zeyuan Chen},
  title = {{CLIFT: Conformal Self-Verification for Web Agent Training and Test-Time Scaling}},
  year = {2026},
  eprint = {2610.06829},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.06829},
}
Figure