CogArena: Benchmarking Multimodal Agents on Interactive Behavioral Experiments

Kianté Fernandez, Caitlin Chen, Jialu Zhou, Bartek Sadowski, Anthony C. Miceli, Ian KrajbicharXiv 2026 · arXiv

TL;DR

CogArena is a new benchmark that evaluates multimodal AI agents on interactive behavioral experiments directly through a web browser, mirroring how human participants engage in research. The study reveals a significant "completion-vs-signature gap": while agents can often finish a task, they frequently fail to replicate the nuanced behavioral "signatures"—such as specific biases or response patterns—that define human cognitive performance.

What Problem Does This Paper Address?

Traditionally, the cognitive abilities of Large Language Models (LLMs) are tested by translating psychological experiments into text-based prompts. However, this "text-only" approach bypasses the visual interfaces, timing constraints, and interactive nature of real-world experiments. For example, a text-based Stroop task only tests language comprehension, whereas the actual visual task also recruits perception and selective attention.

Furthermore, as AI agents become more capable of navigating the web, they pose a growing threat to the validity of online behavioral research. If an agent can "pass" as a human participant, it can contaminate datasets. CogArena aims to provide a more rigorous evaluation of whether AI behavior truly resembles human behavior and to identify which human-like signatures agents can (and cannot) currently reproduce.

Methods

The authors developed a platform that hosts ten classic experimental paradigms (like the Prisoner's Dilemma and Random Dot Motion) implemented in JavaScript. Agents interact with these tasks using a "scaffold"—software that translates the model's text output into browser actions like clicks and keypresses.

The core of CogArena is its three-level scoring pipeline:

  1. L1: Task Completion: Checks if the agent actually finished the experiment and provided valid data.
  2. L2: Performance Accuracy: Compares the agent's raw performance (e.g., reward earned, accuracy) against human baselines.
  3. L3: Behavioral Alignment: Tests for specific "signatures" established in human literature, such as the primacy effect in memory or tit-for-tat reciprocity in social games.
Figure 1: CogArena conceptual architecture. An agent (using any scaffold) interacts with CogArena tasks (behavioral experiments served by the platform), and the resulting trial data are scored at three levels, L1 task completion, L2 performance accuracy, and L3 alignment with task-specific signatures. Trial data are sent to the server during the session, so partial sessions still produce usable data.

Datasets

The paper introduces the CogArena v1 benchmark, which includes 10 curated tasks spanning learning, risk, social decision-making, memory, perception, and moral judgment.

To establish ground truth, the authors derived human baselines from 24 published source studies, using raw participant data wherever possible. They also established a "chance floor" by running a Random Agent that simply presses random keys, providing a baseline for what "zero intelligence" looks like on these tasks.

Key Insights and Ideas

  • Completion $\neq$ Alignment: One of the most striking findings is that finishing an experiment is not evidence of human-like ability. Agents often reached the end of a task but failed to show the behavioral signatures that define human performance.
  • Capacity vs. Bias: The authors distinguish between "capacities" (patterns predicted by normative models, like optimal foraging) and "biases" (patterns that depart from normative models, like moral preferences). Interestingly, agents were much better at reproducing human biases (66% pass rate) than human capacities (31% pass rate).
  • The "Vision" Dependency: By running a "vision off" check (removing screenshots), the researchers found that agent performance dropped by 31–53%. This suggests that agents rely heavily on visual screenshots for interface orientation, even when the task content is primarily text-based.
  • High Intra-Model Variance: The same model on the same task can produce wildly different results across different sessions. This highlights the need for multiple runs (the authors used ten per model-task cell) to get a reliable measure of an agent's "personality" or cognitive profile.

Key Results

The authors evaluated four "economy-tier" multimodal models (Gemini-3-flash-preview, Kimi-k2.5, Qwen3-vl-235b-instruct, and GPT-5.4-nano) and one state-of-the-art model (GPT-5.6-sol).

  • Top Performer: Gemini-3-flash-preview was the most consistent, achieving the highest mean composite score of 58.6.
  • Completion Rates: Completion was far from guaranteed; only 167 out of 400 sessions passed all Level 1 checks. Many agents prematurely declared a task "finished" before it actually was.
  • Signature Success: While agents were excellent at some signatures (e.g., 100% success in preferring to save more lives in moral dilemmas), they failed others completely, such as recognizing previously seen images above chance or showing faster responses at higher motion coherence.
Per-model score distributions across the v1 ten-task set

Limitations and Future Work

The authors acknowledge several key limitations:

  • Human Baseline Gap: Currently, human baselines are drawn from original studies rather than humans performing the CogArena versions of the tasks. Collecting on-platform human data is a top priority for future releases.
  • Model Sample: The evaluation focused primarily on inexpensive "economy-tier" models. Testing a wider range of state-of-the-art models will provide a clearer picture of the current ceiling of AI capabilities.
  • Task Scope: The initial 10 tasks are a curated subset. The authors plan to expand the benchmark to include more phenomena and manipulations.
  • Scaffold Influence: Results describe the model-plus-scaffold combination. Future work could explore how different browser-control tools affect an agent's cognitive performance.

Summary updated

Abstract

Claims about the cognitive capabilities of large language models are typically tested by translating behavioral paradigms into text prompts, bypassing the visual interfaces and time pressure through which human participants actually engage with experiments. At the same time, AI agents can now complete online behavioral tasks and produce human-like data, which raises concerns that online datasets in the social and behavioral sciences are increasingly open to contamination. We introduce CogArena, a benchmark of ten experimental paradigms from the social and behavioral sciences. Agents complete the experiments through a standard web browser, working with the same visual stimuli and response interfaces that online participants use. A three-level scoring procedure evaluates whether the agent completed the task, how accurately it performed, and whether its data show the behavioral signatures each paradigm reliably produces in people. Across four multimodal models, each run ten times per task, we find that finishing an experiment and reproducing its signatures are different things. Many sessions did not finish, and those that did reproduced some signatures reliably and others not at all. CogArena provides the tasks, scoring code, and a public leaderboard for new submissions.
BibTeX
@misc{fernandez2026cogarena,
  author = {Kianté Fernandez and Caitlin Chen and Jialu Zhou and Bartek Sadowski and Anthony C. Miceli and Ian Krajbich},
  title = {{CogArena: Benchmarking Multimodal Agents on Interactive Behavioral Experiments}},
  year = {2026},
  eprint = {2610.36484},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36484},
}
Figure