TL;DR
THEATRON is a new benchmark that moves beyond testing if an LLM can "sound" like a persona to testing if it can consistently act like one over long-term decision-making sequences. By simulating financial trading, the researchers found that while persona prompts change model behavior, they often lead to "stereotyped collapse" (repeating the same action) or "noisy switching" rather than stable, context-sensitive policies.
What Problem Does This Paper Address?
In the world of LLM personalization, we often evaluate models based on their stylistic flair—can a model talk like a pirate or a helpful assistant? However, for high-stakes applications like financial decision-support or social simulation, "persona expression" isn't enough. We need persona persistence.
The authors identify a critical gap: existing benchmarks usually focus on single-turn interactions or identity fidelity in dialogue. They don't measure whether a persona-conditioned model can maintain a coherent, distinguishable strategy over a long horizon where its previous actions affect its future state. If a model claims to be "loss-averse" but starts taking wild risks after a few bad trades, the persona has failed to persist.
Methods
The authors introduce THEATRON, a controlled benchmark using a "financial replay" task.
- The Task: Models act as trading agents over a 102-day market stream. At each step, they receive market evidence (prices, news, SEC filings) and their own account state (previous action and returns).
- Personas: The study uses three behavioral finance personas: Overconfidence, Loss Aversion, and Herding.
- Injection Methods: They test three standard ways to "nudge" the model: Zero-Shot (system prompt only), One-Shot (prompt + one example), and Chain-of-Thought (prompting the model to reason before acting).
- Evaluation Metrics:
- Within-Persona Stability: Do repeated runs of the same persona produce similar results?
- Cross-Persona Separability: Are the trajectories of different personas actually different from each other?
- B/W Ratio: A ratio of between-persona distance to within-persona distance.
- Action Entropy: A measure of how diverse the model's actions are, used to detect "action collapse."
Datasets
THEATRON utilizes the FinAI dataset, specifically focusing on three equities:
- AAPL (Apple): A stable, large-cap asset.
- TSLA (Tesla): A high-volatility, sentiment-sensitive asset.
- BMRN (BioMarin Pharmaceutical): Used for cross-sector replication.
The benchmark covers 102 trading days (Jan–May 2025) and includes adjusted closing prices, financial news summaries, and SEC filing passages (10-K/10-Q).
Key Insights and Ideas
- Separation $\neq$ Persistence: A model might produce very different results for "Overconfident" vs. "Loss Averse" prompts, but that doesn't mean it's following a coherent policy. The authors found that high separation often comes from stereotyped policy reduction—where the model simply maps a persona to a single repeated action (e.g., always BUY) regardless of market conditions.
- The "Noisy Switching" Failure: Conversely, some models show high action diversity but fail to maintain a stable identity across different runs of the same scenario. They react to local market cues but lack "temporal state binding."
- Prompting Scaffolding is a Double-Edged Sword: While One-Shot and Chain-of-Thought (CoT) prompting significantly change behavior, they don't consistently fix stability issues. In some cases, they even exacerbate the collapse into deterministic action priors.
- Comparison to Existing Benchmarks: As shown in the table below, THEATRON is unique in its support for sequential decisions combined with rigorous stability and separability metrics.
Key Results
The study evaluated four major models: GPT-5.4-mini, Gemini-2.5-Flash, Qwen3-8B, and Qwen3-14B.
- Behavioral Shifts: Persona conditioning is consequential. For example, the no-persona GPT policy chose "HOLD" 90% of the time on AAPL, but the "Overconfidence" persona increased non-HOLD actions by up to 96.5 percentage points.
- Gemini's Collapse: Gemini-2.5-Flash under the "Overconfidence" persona repeatedly collapsed to a "BUY" action prior on TSLA, resulting in zero return variance and zero action entropy across all five test runs.
- GPT's Instability: GPT-5.4-mini showed the opposite; under "Overconfidence," its final returns on AAPL ranged from -27.26% to +11.73% across identical runs, indicating a lack of stable policy identity.
Limitations and Future Work
The authors acknowledge several limitations:
- Simplified Trading: The task uses a basic BUY/HOLD/SELL setup and ignores position sizing, transaction costs, and liquidity.
- Limited Asset Scope: While they tested multiple stocks, the evaluation is still centered on a specific financial window.
- SFT Failure: A small-scale Supervised Fine-Tuning (SFT) experiment on Qwen3-8B failed to resolve the stability-separability tension, suggesting that simple imitation learning isn't enough to instill persistent personas.
Future Work: The authors suggest that future research should focus on optimizing persona persistence at the trajectory level, ensuring models are both context-sensitive and behaviorally consistent over time.
Summary updated