BehaviorBench: Modeling Real-World User Decisions from Behavioral Traces

Liangwei Yang, Jielin Qiu, Zixiang Chen, Ming Zhu, Juntao Tan, Zhiwei Liu, Wenting Zhao, Zhujun Lan, Akshara Prabhakar, Silvio Savarese, Huan Wang, Shelby HeineckearXiv 2026 · arXiv

TL;DR

BehaviorBench is a new evaluation benchmark that tests an AI's ability to predict user decisions using real-world, on-chain behavioral traces from prediction markets rather than synthetic personas. The benchmark reveals a key dichotomy in personalization: predicting a user's high-level beliefs requires abstract profile summaries, whereas predicting granular, transaction-level trades demands concrete sequential history.

What Problem Does This Paper Address?

As AI agents transition from general-purpose tools to deeply embedded decision-support assistants, they must learn to understand the specific individuals they serve. A truly personalized agent should infer a user's beliefs, preferences, and tendencies from their prior actions rather than applying a generic rule to everyone.

However, evaluating this capability is notoriously difficult. Existing personalization benchmarks often rely on explicit text personas, static survey responses, or simulated user behavior generated by other LLMs. Recent research cautions that these synthetic simulations frequently diverge from actual human behavior. Furthermore, real-world human preferences are rarely stated explicitly; they are hidden implicitly within long sequences of actions—what a person repeatedly chooses, modifies, or abandons.

To bridge this "sim-to-real" gap, the authors introduce BehaviorBench, a benchmark grounded entirely in observed, real-world behavioral traces.

Methods

The authors leverage public event-market data to reconstruct high-fidelity, longitudinal decision histories. They design a quality-controlled pipeline to process raw transaction logs into clean, chronological user trajectories.

Figure 1: Overview of the BEHAVIORBENCH construction pipeline. We collect public event metadata and on-chain logs, reconstruct wallet–market–outcome behavior, apply cohort and quality-control filters, and build two benchmark layers for Belief prediction and Trade prediction with disjoint support pools for retrieval-based evaluation.

BehaviorBench evaluates user modeling at two distinct levels of abstraction:

  1. Belief Prediction: This layer aggregates a user's transaction sequence within a specific market to predict their final revealed stance (e.g., a YES or NO position) and their confidence level (the concentration of their capital on that outcome).
  2. Trade Prediction: This layer focuses on local, short-horizon actions, tasking the model with predicting the direction (BUY/SELL) and exact token amount of the very next transaction.

To understand how models process historical evidence, the authors evaluate them across four distinct history interfaces:

  • No Personalization: The baseline condition, providing only the target market context.
  • DirectGen: Provides a recency-truncated window of the user's raw transaction history.
  • ProfileGen: Replaces raw history with an LLM-generated structured profile summarizing the user's topic preferences, decision style, and consistency patterns.
  • RetrievalGen: Uses a Retrieval-Augmented Generation (RAG) approach to supply behavioral evidence from similar, disjoint "support wallets."

Datasets

The benchmark is built using public metadata from the Polymarket API and on-chain transaction logs from the Polygon blockchain. The data vintage ends in December 2025 to avoid a massive surge in transaction volume in early 2026, which likely represents highly automated bot activity.

Across 2,000 unique evaluation wallets stratified into distinct activity buckets, BehaviorBench contains:

  • 141,445 Belief instances
  • 1,485,972 Trade instances
  • A disjoint support pool of 11,676 wallets dedicated to retrieval-based evaluation.

Key Insights and Ideas

  • Personalization is Not a Monolithic Capability: The optimal way to represent a user's history depends entirely on the target task. High-level belief prediction is best served by abstract, compressed user profiles (ProfileGen). Conversely, transaction-level trade prediction requires fine-grained, raw sequential context (DirectGen); compressing this history into a profile actually hurts performance.
  • The Leaderboard Flips Based on the Metric: Frontier models (like GPT-5) dominate directional metrics like choice and trade accuracy. However, open-weight models (like Gemma 3) are highly competitive—and sometimes superior—on continuous error metrics, such as predicting trade magnitudes and calibrating confidence scores.
  • More History Can Degrade Calibration: In history-scaling experiments, providing a modest history window sharply improves belief accuracy. However, exposing an unbounded history to profile generators can eventually degrade confidence calibration, likely because the model struggles to compress highly heterogeneous behavior into a static schema.
Belief vs. Trade accuracy decoupling across different models and methods.

Key Results

  • Clear Personalization Gains: History-aware models consistently outperform the non-personalized baseline. For instance, on Belief prediction, GPT-5's choice accuracy jumps from a baseline of 53.13% to 73.20% when augmented with ProfileGen.
  • Trade Granularity Challenges: For Trade prediction, DirectGen yields the highest direction accuracy for frontier models (GPT-5 achieves 75.38%). Using ProfileGen or RetrievalGen causes a noticeable drop in accuracy, underscoring that local actions are highly dependent on immediate context rather than generalized user traits.
  • Substantial Room for Improvement: Even the strongest frontier models perform well below the ceiling on continuous metrics, leaving a large headroom gap for future research in user modeling and calibration.

Limitations and Future Work

The authors note several critical boundaries to this work:

  • Pseudonymous Accounts: On-chain wallets are behavioral accounts, not verified individuals. A single human might operate multiple wallets, or a single wallet might be run by an automated trading bot or a collective group.
  • Observability Limits: A model cannot fully observe a user's hidden constraints, such as total portfolio liquidity, transaction fees, or broader off-chain hedging strategies, which heavily dictate trade amounts.
  • Broader Impact Risks: The user-modeling techniques evaluated here could potentially be repurposed for malicious targeting, financial manipulation, or intrusive surveillance.

Future work includes exploring adaptive history interfaces that can dynamically toggle between raw sequential context and abstract profiles depending on the nature of the decision support task.

Summary updated

Abstract

Many decision-support settings require systems that adapt to individual users, but evaluation data for this problem remain limited. Existing benchmarks for user understanding often rely on simulated users or model-generated behavior, even though recent work cautions that model-based simulations can diverge systematically from human behavior. We introduce \textsc{BehaviorBench}, a benchmark for evaluating personalized decision modeling from real-world behavioral traces. \textsc{BehaviorBench} reconstructs wallet-level decision histories from observed public prediction-market and on-chain records, and organizes them into two complementary task layers: \emph{Belief prediction}, which predicts a user's final revealed stance and confidence in a market, and \emph{Trade prediction}, which predicts the direction and amount of individual transactions. Across 2,000 evaluation wallets, the benchmark contains 141,445 Belief instances and 1,485,972 Trade instances, with disjoint support pools for retrieval-based evaluation. We evaluate frontier and open-weight generative models under four history interfaces: no personalization, direct recent history, generated user profiles, and retrieved support-wallet evidence. Personalization improves Belief prediction more consistently than Trade prediction, model rankings change across task layers and metrics, and different history interfaces expose different failure modes. \textsc{BehaviorBench} provides an evaluation setting for studying whether personalized methods can use real-world behavioral evidence rather than simulated users alone.
BibTeX
@misc{yang2026behaviorbench,
  author = {Liangwei Yang and Jielin Qiu and Zixiang Chen and Ming Zhu and Juntao Tan and Zhiwei Liu and Wenting Zhao and Zhujun Lan and Akshara Prabhakar and Silvio Savarese and Huan Wang and Shelby Heinecke},
  title = {{BehaviorBench: Modeling Real-World User Decisions from Behavioral Traces}},
  year = {2026},
  eprint = {2606.02798},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.02798},
}
Figure