TL;DR
ClawArena is a new benchmark designed to evaluate AI agents in realistic, evolving information environments where data is scattered, contradictory, and updated over time. By introducing a 14-category taxonomy across three core dimensions—multi-source conflict reasoning, dynamic belief revision, and implicit personalization—it challenges agents to maintain an accurate worldview. Evaluation of 18 language models across 5 agent frameworks reveals that while model capability dominates, framework design and update strategies drastically alter an agent's reliability.
What Problem Does This Paper Address?
Most existing AI agent benchmarks operate under a major simplification: they assume a static, single-authority environment. In these setups, an agent is given a fixed set of files or a single database, and the ground truth never changes.
In the real world, persistent AI assistants face a much messier reality. Information is fragmented across multiple channels (e.g., Slack, emails, system logs), sources frequently contradict one another, new updates can completely invalidate previous conclusions, and user preferences are rarely explicitly stated up front. To be truly useful, an agent must be able to adjudicate conflicts, revise its beliefs dynamically, and pick up on implicit user preferences over time. ClawArena addresses this gap by providing an evaluation ecosystem built around these exact challenges.
Methods
ClawArena structures its evaluation around a six-layer specification system. Layer 0 represents a hidden, absolute ground truth, while Layers 1–4 are visible to the agent as workspace files, multi-channel session histories, evaluation questions, and staged update packages.
The evaluation tests three coupled dimensions:
- Multi-source conflict reasoning (MS): Forcing the agent to judge source reliability when different channels report conflicting information.
- Dynamic belief revision (DU): Injecting new evidence over time that forces the agent to explicitly revise prior conclusions.
- Implicit personalization (P): Requiring the agent to learn and apply user formatting or stylistic preferences purely from behavioral feedback, tested later in "silent-exam" rounds without reminders.
To assess performance, ClawArena uses two question formats: multi-choice (set-selection) to test high-level reasoning, and shell-based executable checks to verify actual workspace file grounding.
Crucially, the authors introduce the Composite Reliability Score (CRS) to measure behavioral consistency beyond raw accuracy. CRS combines the standard Task Completion Rate (TCR) with a Robustness metric. This Robustness metric decomposes into Success Cohesion (rewarding consecutive success streaks) and Failure Dispersion (penalizing prolonged, clustered collapses).
Datasets
The authors introduced CLAWARENA, a benchmark consisting of 12 multi-turn scenarios spanning 337 evaluation rounds and 45 dynamic updates. The scenarios cover diverse professional domains, including retail analytics, finance, healthcare, HR, and research integrity.
To ensure authenticity, the dataset generation pipeline is grounded in over 200 real-world empirical distributions tracking email volume, git commit patterns, and social network structures. This ensures realistic message timing, contact frequencies, and noise levels (where 30–50% of background messages are irrelevant chatter).
Key Insights and Ideas
- Model Capability vs. Framework Design: Model capability is the primary driver of performance, accounting for a 29-point CRS range across models. However, framework design is nearly as critical, introducing up to a 24-point performance swing for the exact same model.
- Asymmetric Belief Revision: An agent's ability to revise its beliefs depends on how an update is designed, not the volume of data. Agents handle updates that simply extend prior evidence well, but frequently fail when an update requires a "bridging inference" that forces them to re-interpret older claims.
- Brittle Personalization: Passing a lenient preference check early in a scenario does not guarantee an agent will maintain that preference when later rounds introduce stricter constraints. Preference tracking remains brittle rather than deeply internalized.
- Independence of Reasoning and Grounding: Performance on multiple-choice reasoning and shell-based workspace grounding are only moderately correlated. An agent can be excellent at high-level logic choices while completely failing at literal structural compliance and file parsing.
Key Results
The benchmark evaluated 18 language models across 5 frameworks (OpenClaw, Claude Code, NanoBot, PicoClaw, and MetaClaw).
- Top Performers: Among proprietary models, GPT-5.5 led the pack with a CRS of 68.28. In the open-weight category, Gemma-4-31B achieved a highly competitive CRS of 63.80, even dominating the multi-choice reasoning sub-score (81.05%) across all models.
- Framework Mismatches: Cross-family framework pairings can cause severe degradation. For example, when the OpenAI GPT family was evaluated inside Anthropic's Claude Code harness, performance collapsed significantly (e.g., GPT-5.5 dropped to 44.50 CRS), highlighting how tightly optimized harnesses are to specific instruction-following styles.
- MetaClaw Skill Overlay: MetaClaw’s skill-driven self-evolution mechanism—which injects procedural skills derived from past failures directly into the prompt—reliably improved Robustness across all tested model families (e.g., providing a +1.19 CRS boost for GLM-5.1) without harming raw accuracy.
Limitations and Future Work
The primary limitation of ClawArena is its reliance on staged updates, meaning new information is injected at pre-determined evaluation rounds. The authors note that a natural and necessary extension of this work is moving toward live, unconstrained environments. In such environments, agents will not just react to incoming streams but must actively formulate their own queries against real-time, live information sources to maintain their beliefs.
Summary updated