Abstract
Large Language Models (LLMs) often struggle with "Theory of Mind"—the ability to infer implicit mental states and social nuances. This paper introduces StimuliQA, a high-quality dataset of expert-labeled psychological narratives, and Psy-Interpreter, a reinforcement learning framework that uses "Bilateral Rewards" to teach compact models (0.5B–3B parameters) how to reason like human experts. By rewarding concise reasoning for simple tasks and analytical depth for complex ones, Psy-Interpreter enables small models to outperform much larger proprietary models like GPT-4 in social-cognitive accuracy.
Introduction
While LLMs are linguistic powerhouses, they frequently fail at tasks requiring deep psychological reasoning, such as empathy, social common sense, and inferring emotions in ambiguous contexts. Current training methods face three core hurdles: a scarcity of nuanced human-labeled data, a "one-size-fits-all" reasoning style that ignores task complexity, and the tendency of compact models to overfit on surface-level cues rather than internalizing robust social-cognitive principles.
The StimuliQA Dataset
To address the data gap, the authors developed StimuliQA, a landmark dataset for psychological reasoning. It contains 3,280 real-life narratives labeled by psychology experts across 58 variables. These variables cover emotional reactions, redemptive arcs, and collective psychology (e.g., social cohesion), resulting in over 15,000 high-quality QA pairs.
The Psy-Interpreter Framework
The authors propose Psy-Interpreter, a framework built on two novel pillars designed to mimic human cognitive processes:
Bilateral Reinforcement Learning (BRL)
Inspired by dual-process theory (System 1 vs. System 2 thinking), BRL uses a Bilateral Reward function. Instead of just rewarding "correctness," it rewards the model for being concise on simple tasks and detailed on complex ones—but only if the final answer is accurate. This prevents "reward hacking" and ensures the model develops appropriate reasoning depth for the context.
Trajectory-aware GRPO (T-GRPO)
Building on Group Relative Policy Optimization, the authors added a Trajectory Cache. This tracks the model's recent performance to stabilize rewards and adjust them based on trends, making the training process more robust across different model scales.
Key Results and Insights
The Psy-Interpreter framework demonstrates that expert labels and adaptive reasoning significantly enhance model intelligence:
- Superior Performance: The Psy-Interpreter (3B) model achieved an F1 score of 28.17 on ToMbench, surpassing GPT-4 nano (22.11) and Claude 3 Haiku (18.81).
- Zero-Shot Generalization: Despite never seeing the SocialIQa dataset during training, the 3B model reached an F1 of 82.82, showcasing incredible transferability of psychological principles.
- Small Models, Big Brains: Compact models (0.5B–3B) trained with this framework can internalize complex social-cognitive principles that previously required massive parameter counts.
- Qualitative Depth: In case studies, the Bilateral model correctly identified subtle emotions like "worry" by suppressing irrelevant surface-level inferences (like "pride") that standard models often hallucinate.
Conclusion and Future Work
The success of Psy-Interpreter proves that human nuance in psychology is currently irreplaceable for training robust social AI. Future work will focus on refining reward signals to incorporate fine-grained psychological metrics, such as causal consistency and emotional salience, and expanding the framework to handle complex moral judgments.
Summary updated