TL;DR
The authors introduce ToMStories, a new evaluation dataset of ~6,000 synthetic narrative vignettes designed to test if Large Language Models (LLMs) can infer character psychology from subtext. While the largest models (like Llama-3.1-70B) can match or exceed human performance on multiple-choice questions, they rely on simplified behavioral "shortcuts" and struggle with open-ended reasoning, suggesting that current benchmarks may overestimate how well models truly understand human social nuances.
What Problem Does This Paper Address?
Theory-of-Mind (ToM)—the ability to infer others' beliefs, feelings, and motivations—is essential for AI to act as a truly helpful conversational companion. However, most existing ToM benchmarks focus on tracking simple beliefs (e.g., "Where does Sally think the ball is?") rather than complex emotional states or motives.
Furthermore, human communication is often indirect; we "show" rather than "tell." This paper investigates whether LLMs can handle subtext—meaning communicated implicitly through behavior and description—and explores the internal mechanisms models use to make these psychological inferences.
Methods
The researchers developed a pipeline to generate controlled synthetic data using Claude Sonnet 4.6. For every scenario, they created three variants:
- Implicit: The psychological state is conveyed only through subtext (behavior/description).
- Explicit: The state is stated directly or through unambiguous physical reactions.
- Conflicted: Spoken dialogue suggests one state (e.g., excitement), while description suggests another (e.g., hurt).
To look "under the hood," the authors evaluated 13 open-weight models (1B to 72B parameters) using:
- Linear Probes: Training simple classifiers on the models' internal hidden states to see if psychological information is "linearly decodable."
- Steering Vectors: Extracting the mathematical direction in activation space that represents "implicitness" and using it to manipulate model outputs.
- Indicator Phrase Analysis: Asking models to list behavioral cues (e.g., "avoiding eye contact") associated with specific emotions to see if models share a common "social vocabulary."
Datasets
The primary contribution is the ToMStories dataset. It contains ~6,000 vignettes covering four psychological categories: Emotions, Fears, Motivations, and Emotional Wounds. The scenarios are grounded in professional writing resources (Thesaurus Description Database for writers) to ensure a wide and realistic range of human experiences.
Key Insights and Ideas
- The "Indicator Phrase" Shortcut: Models seem to rely on a shared set of behavioral cues to infer internal states. Interestingly, the best predictor of a model's performance on ToMStories wasn't its size, but how much its "vocabulary" of behavioral cues overlapped with the model that generated the data (Claude). This suggests models might be "shortcut-matching" rather than deeply reasoning.
- Implicit = Muted: By using steering vectors, the authors found that for LLMs, "implicit" expression is internally represented as muted emotion. Steering a model toward an implicit style consistently reduced the intensity, valence, and arousal of the emotional words it used.
- Dialogue Bias: In "conflicted" scenarios where a character's words contradict their actions, models tend to trust the dialogue more than humans do. While humans split their interpretations more evenly, models showed a stronger bias toward the explicit spoken word.
Key Results
- Multiple Choice: The largest models matched or beat the human baseline. Llama-3.1-70B achieved 88.4% accuracy, compared to the human baseline of 80.3%.
- Open-Ended Reasoning: Models still lag behind humans when they have to generate their own interpretations. The best model (Qwen-2.5-72B) reached 58.6% accuracy, while humans scored 64.2%.
- Correlation: There is a massive correlation (r = 0.89 to 0.92) between a model's performance and its "indicator phrase overlap" with the data generator. This highlights a potential "synthetic data bias" where models appear smarter because they share the same simplified social logic as the generator.
Limitations and Future Work
The authors acknowledge that using a single model (Claude) to generate the entire dataset is a major limitation; the benchmark might be measuring how well models align with Claude's specific social conventions rather than general human psychology. The cultural scope is also limited to Western, English-language framing.
Future work should involve human-written narratives and vignettes from multiple different LLM generators to decouple "alignment with the generator" from true interpretive ability. Additionally, the authors suggest investigating whether these "indicator phrases" causally shape how models interpret social situations in real-time.
Summary updated