TL;DR
BehaviorBench is a new evaluation benchmark that tests an AI's ability to predict user decisions using real-world, on-chain behavioral traces from prediction markets rather than synthetic personas. The benchmark reveals a key dichotomy in personalization: predicting a user's high-level beliefs requires abstract profile summaries, whereas predicting granular, transaction-level trades demands concrete sequential history.
What Problem Does This Paper Address?
As AI agents transition from general-purpose tools to deeply embedded decision-support assistants, they must learn to understand the specific individuals they serve. A truly personalized agent should infer a user's beliefs, preferences, and tendencies from their prior actions rather than applying a generic rule to everyone.
However, evaluating this capability is notoriously difficult. Existing personalization benchmarks often rely on explicit text personas, static survey responses, or simulated user behavior generated by other LLMs. Recent research cautions that these synthetic simulations frequently diverge from actual human behavior. Furthermore, real-world human preferences are rarely stated explicitly; they are hidden implicitly within long sequences of actions—what a person repeatedly chooses, modifies, or abandons.
To bridge this "sim-to-real" gap, the authors introduce BehaviorBench, a benchmark grounded entirely in observed, real-world behavioral traces.
Methods
The authors leverage public event-market data to reconstruct high-fidelity, longitudinal decision histories. They design a quality-controlled pipeline to process raw transaction logs into clean, chronological user trajectories.
BehaviorBench evaluates user modeling at two distinct levels of abstraction:
- Belief Prediction: This layer aggregates a user's transaction sequence within a specific market to predict their final revealed stance (e.g., a YES or NO position) and their confidence level (the concentration of their capital on that outcome).
- Trade Prediction: This layer focuses on local, short-horizon actions, tasking the model with predicting the direction (BUY/SELL) and exact token amount of the very next transaction.
To understand how models process historical evidence, the authors evaluate them across four distinct history interfaces:
- No Personalization: The baseline condition, providing only the target market context.
- DirectGen: Provides a recency-truncated window of the user's raw transaction history.
- ProfileGen: Replaces raw history with an LLM-generated structured profile summarizing the user's topic preferences, decision style, and consistency patterns.
- RetrievalGen: Uses a Retrieval-Augmented Generation (RAG) approach to supply behavioral evidence from similar, disjoint "support wallets."
Datasets
The benchmark is built using public metadata from the Polymarket API and on-chain transaction logs from the Polygon blockchain. The data vintage ends in December 2025 to avoid a massive surge in transaction volume in early 2026, which likely represents highly automated bot activity.
Across 2,000 unique evaluation wallets stratified into distinct activity buckets, BehaviorBench contains:
- 141,445 Belief instances
- 1,485,972 Trade instances
- A disjoint support pool of 11,676 wallets dedicated to retrieval-based evaluation.
Key Insights and Ideas
- Personalization is Not a Monolithic Capability: The optimal way to represent a user's history depends entirely on the target task. High-level belief prediction is best served by abstract, compressed user profiles (
ProfileGen). Conversely, transaction-level trade prediction requires fine-grained, raw sequential context (DirectGen); compressing this history into a profile actually hurts performance. - The Leaderboard Flips Based on the Metric: Frontier models (like GPT-5) dominate directional metrics like choice and trade accuracy. However, open-weight models (like Gemma 3) are highly competitive—and sometimes superior—on continuous error metrics, such as predicting trade magnitudes and calibrating confidence scores.
- More History Can Degrade Calibration: In history-scaling experiments, providing a modest history window sharply improves belief accuracy. However, exposing an unbounded history to profile generators can eventually degrade confidence calibration, likely because the model struggles to compress highly heterogeneous behavior into a static schema.
Key Results
- Clear Personalization Gains: History-aware models consistently outperform the non-personalized baseline. For instance, on Belief prediction, GPT-5's choice accuracy jumps from a baseline of 53.13% to 73.20% when augmented with
ProfileGen. - Trade Granularity Challenges: For Trade prediction,
DirectGenyields the highest direction accuracy for frontier models (GPT-5 achieves 75.38%). UsingProfileGenorRetrievalGencauses a noticeable drop in accuracy, underscoring that local actions are highly dependent on immediate context rather than generalized user traits. - Substantial Room for Improvement: Even the strongest frontier models perform well below the ceiling on continuous metrics, leaving a large headroom gap for future research in user modeling and calibration.
Limitations and Future Work
The authors note several critical boundaries to this work:
- Pseudonymous Accounts: On-chain wallets are behavioral accounts, not verified individuals. A single human might operate multiple wallets, or a single wallet might be run by an automated trading bot or a collective group.
- Observability Limits: A model cannot fully observe a user's hidden constraints, such as total portfolio liquidity, transaction fees, or broader off-chain hedging strategies, which heavily dictate trade amounts.
- Broader Impact Risks: The user-modeling techniques evaluated here could potentially be repurposed for malicious targeting, financial manipulation, or intrusive surveillance.
Future work includes exploring adaptive history interfaces that can dynamically toggle between raw sequential context and abstract profiles depending on the nature of the decision support task.
Summary updated