TL;DR
CogArena is a new benchmark that evaluates multimodal AI agents on interactive behavioral experiments directly through a web browser, mirroring how human participants engage in research. The study reveals a significant "completion-vs-signature gap": while agents can often finish a task, they frequently fail to replicate the nuanced behavioral "signatures"—such as specific biases or response patterns—that define human cognitive performance.
What Problem Does This Paper Address?
Traditionally, the cognitive abilities of Large Language Models (LLMs) are tested by translating psychological experiments into text-based prompts. However, this "text-only" approach bypasses the visual interfaces, timing constraints, and interactive nature of real-world experiments. For example, a text-based Stroop task only tests language comprehension, whereas the actual visual task also recruits perception and selective attention.
Furthermore, as AI agents become more capable of navigating the web, they pose a growing threat to the validity of online behavioral research. If an agent can "pass" as a human participant, it can contaminate datasets. CogArena aims to provide a more rigorous evaluation of whether AI behavior truly resembles human behavior and to identify which human-like signatures agents can (and cannot) currently reproduce.
Methods
The authors developed a platform that hosts ten classic experimental paradigms (like the Prisoner's Dilemma and Random Dot Motion) implemented in JavaScript. Agents interact with these tasks using a "scaffold"—software that translates the model's text output into browser actions like clicks and keypresses.
The core of CogArena is its three-level scoring pipeline:
- L1: Task Completion: Checks if the agent actually finished the experiment and provided valid data.
- L2: Performance Accuracy: Compares the agent's raw performance (e.g., reward earned, accuracy) against human baselines.
- L3: Behavioral Alignment: Tests for specific "signatures" established in human literature, such as the primacy effect in memory or tit-for-tat reciprocity in social games.
Datasets
The paper introduces the CogArena v1 benchmark, which includes 10 curated tasks spanning learning, risk, social decision-making, memory, perception, and moral judgment.
To establish ground truth, the authors derived human baselines from 24 published source studies, using raw participant data wherever possible. They also established a "chance floor" by running a Random Agent that simply presses random keys, providing a baseline for what "zero intelligence" looks like on these tasks.
Key Insights and Ideas
- Completion $\neq$ Alignment: One of the most striking findings is that finishing an experiment is not evidence of human-like ability. Agents often reached the end of a task but failed to show the behavioral signatures that define human performance.
- Capacity vs. Bias: The authors distinguish between "capacities" (patterns predicted by normative models, like optimal foraging) and "biases" (patterns that depart from normative models, like moral preferences). Interestingly, agents were much better at reproducing human biases (66% pass rate) than human capacities (31% pass rate).
- The "Vision" Dependency: By running a "vision off" check (removing screenshots), the researchers found that agent performance dropped by 31–53%. This suggests that agents rely heavily on visual screenshots for interface orientation, even when the task content is primarily text-based.
- High Intra-Model Variance: The same model on the same task can produce wildly different results across different sessions. This highlights the need for multiple runs (the authors used ten per model-task cell) to get a reliable measure of an agent's "personality" or cognitive profile.
Key Results
The authors evaluated four "economy-tier" multimodal models (Gemini-3-flash-preview, Kimi-k2.5, Qwen3-vl-235b-instruct, and GPT-5.4-nano) and one state-of-the-art model (GPT-5.6-sol).
- Top Performer: Gemini-3-flash-preview was the most consistent, achieving the highest mean composite score of 58.6.
- Completion Rates: Completion was far from guaranteed; only 167 out of 400 sessions passed all Level 1 checks. Many agents prematurely declared a task "finished" before it actually was.
- Signature Success: While agents were excellent at some signatures (e.g., 100% success in preferring to save more lives in moral dilemmas), they failed others completely, such as recognizing previously seen images above chance or showing faster responses at higher motion coherence.
Limitations and Future Work
The authors acknowledge several key limitations:
- Human Baseline Gap: Currently, human baselines are drawn from original studies rather than humans performing the CogArena versions of the tasks. Collecting on-platform human data is a top priority for future releases.
- Model Sample: The evaluation focused primarily on inexpensive "economy-tier" models. Testing a wider range of state-of-the-art models will provide a clearer picture of the current ceiling of AI capabilities.
- Task Scope: The initial 10 tasks are a curated subset. The authors plan to expand the benchmark to include more phenomena and manipulations.
- Scaffold Influence: Results describe the model-plus-scaffold combination. Future work could explore how different browser-control tools affect an agent's cognitive performance.
Summary updated