ClawArena: Benchmarking AI Agents in Evolving Information Environments

Haonian Ji, Kaiwen Xiong, Siwei Han, Peng Xia, Shi Qiu, Yiyang Zhou, Jiaqi Liu, Jinlong Li, Bingzhou Li, Zeyu Zheng, Cihang Xie, Huaxiu YaoarXiv 2026 · arXiv

TL;DR

ClawArena is a new benchmark designed to evaluate AI agents in realistic, evolving information environments where data is scattered, contradictory, and updated over time. By introducing a 14-category taxonomy across three core dimensions—multi-source conflict reasoning, dynamic belief revision, and implicit personalization—it challenges agents to maintain an accurate worldview. Evaluation of 18 language models across 5 agent frameworks reveals that while model capability dominates, framework design and update strategies drastically alter an agent's reliability.

What Problem Does This Paper Address?

Most existing AI agent benchmarks operate under a major simplification: they assume a static, single-authority environment. In these setups, an agent is given a fixed set of files or a single database, and the ground truth never changes.

In the real world, persistent AI assistants face a much messier reality. Information is fragmented across multiple channels (e.g., Slack, emails, system logs), sources frequently contradict one another, new updates can completely invalidate previous conclusions, and user preferences are rarely explicitly stated up front. To be truly useful, an agent must be able to adjudicate conflicts, revise its beliefs dynamically, and pick up on implicit user preferences over time. ClawArena addresses this gap by providing an evaluation ecosystem built around these exact challenges.

Figure 1: Overview of CLAWARENA. An evolving information environment with conflicting and progressively updated evidence is structured along three coupled evaluation dimen- sions, instantiated through two complementary question formats, and summarized by a composite reliability score that jointly reflects accuracy and behavioural consistency.

Methods

ClawArena structures its evaluation around a six-layer specification system. Layer 0 represents a hidden, absolute ground truth, while Layers 1–4 are visible to the agent as workspace files, multi-channel session histories, evaluation questions, and staged update packages.

The evaluation tests three coupled dimensions:

  1. Multi-source conflict reasoning (MS): Forcing the agent to judge source reliability when different channels report conflicting information.
  2. Dynamic belief revision (DU): Injecting new evidence over time that forces the agent to explicitly revise prior conclusions.
  3. Implicit personalization (P): Requiring the agent to learn and apply user formatting or stylistic preferences purely from behavioral feedback, tested later in "silent-exam" rounds without reminders.

To assess performance, ClawArena uses two question formats: multi-choice (set-selection) to test high-level reasoning, and shell-based executable checks to verify actual workspace file grounding.

Crucially, the authors introduce the Composite Reliability Score (CRS) to measure behavioral consistency beyond raw accuracy. CRS combines the standard Task Completion Rate (TCR) with a Robustness metric. This Robustness metric decomposes into Success Cohesion (rewarding consecutive success streaks) and Failure Dispersion (penalizing prolonged, clustered collapses).

Datasets

The authors introduced CLAWARENA, a benchmark consisting of 12 multi-turn scenarios spanning 337 evaluation rounds and 45 dynamic updates. The scenarios cover diverse professional domains, including retail analytics, finance, healthcare, HR, and research integrity.

The three-stage construction pipeline used to generate ClawArena scenarios based on real-world empirical distributions.

To ensure authenticity, the dataset generation pipeline is grounded in over 200 real-world empirical distributions tracking email volume, git commit patterns, and social network structures. This ensures realistic message timing, contact frequencies, and noise levels (where 30–50% of background messages are irrelevant chatter).

Key Insights and Ideas

  • Model Capability vs. Framework Design: Model capability is the primary driver of performance, accounting for a 29-point CRS range across models. However, framework design is nearly as critical, introducing up to a 24-point performance swing for the exact same model.
  • Asymmetric Belief Revision: An agent's ability to revise its beliefs depends on how an update is designed, not the volume of data. Agents handle updates that simply extend prior evidence well, but frequently fail when an update requires a "bridging inference" that forces them to re-interpret older claims.
  • Brittle Personalization: Passing a lenient preference check early in a scenario does not guarantee an agent will maintain that preference when later rounds introduce stricter constraints. Preference tracking remains brittle rather than deeply internalized.
  • Independence of Reasoning and Grounding: Performance on multiple-choice reasoning and shell-based workspace grounding are only moderately correlated. An agent can be excellent at high-level logic choices while completely failing at literal structural compliance and file parsing.

Key Results

The benchmark evaluated 18 language models across 5 frameworks (OpenClaw, Claude Code, NanoBot, PicoClaw, and MetaClaw).

  • Top Performers: Among proprietary models, GPT-5.5 led the pack with a CRS of 68.28. In the open-weight category, Gemma-4-31B achieved a highly competitive CRS of 63.80, even dominating the multi-choice reasoning sub-score (81.05%) across all models.
  • Framework Mismatches: Cross-family framework pairings can cause severe degradation. For example, when the OpenAI GPT family was evaluated inside Anthropic's Claude Code harness, performance collapsed significantly (e.g., GPT-5.5 dropped to 44.50 CRS), highlighting how tightly optimized harnesses are to specific instruction-following styles.
  • MetaClaw Skill Overlay: MetaClaw’s skill-driven self-evolution mechanism—which injects procedural skills derived from past failures directly into the prompt—reliably improved Robustness across all tested model families (e.g., providing a +1.19 CRS boost for GLM-5.1) without harming raw accuracy.

Limitations and Future Work

The primary limitation of ClawArena is its reliance on staged updates, meaning new information is injected at pre-determined evaluation rounds. The authors note that a natural and necessary extension of this work is moving toward live, unconstrained environments. In such environments, agents will not just react to incoming streams but must actively formulate their own queries against real-time, live information sources to maintain their beliefs.

Summary updated

Abstract

AI agents deployed as persistent assistants must maintain correct beliefs as their information environment evolves. In practice, evidence is scattered across heterogeneous sources that often contradict one another, new information can invalidate earlier conclusions, and user preferences surface through corrections rather than explicit instructions. Existing benchmarks largely assume static, single-authority settings and do not evaluate whether agents can keep up with this complexity. We introduce ClawArena, a benchmark for evaluating AI agents in evolving information environments. Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces across multi-channel sessions, workspace files, and staged updates. Evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization, whose interactions yield a 14-category question taxonomy. Two question formats, multi-choice (set-selection) and shell-based executable checks, test both reasoning and workspace grounding. ClawArena comprises 12 multi-turn scenarios spanning 337 evaluation rounds with 45 dynamic updates, evaluated across five agent frameworks and 18 language models from proprietary, community-accessible, and self-hosted sources. Experiments show that model capability accounts for a 29-point score range across models while framework design accounts for up to a 24-point range, that MetaClaw's skill overlay reliably improves score without degrading accuracy, and that belief revision difficulty is determined by update design strategy rather than update volume. Code is available at https://github.com/aiming-lab/ClawArena.
BibTeX
@misc{ji2026clawarena,
  author = {Haonian Ji and Kaiwen Xiong and Siwei Han and Peng Xia and Shi Qiu and Yiyang Zhou and Jiaqi Liu and Jinlong Li and Bingzhou Li and Zeyu Zheng and Cihang Xie and Huaxiu Yao},
  title = {{ClawArena: Benchmarking AI Agents in Evolving Information Environments}},
  year = {2026},
  eprint = {2604.04202},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.04202},
}
Figure