Large Emotional World Model

Yazhou Zhang, Changhao Song, Yazhou Zhang 0001, Hui Gao, Chang Yang, Peng ZhangarXiv 2025 · DOI · arXiv

Abstract

While contemporary world models like Sora demonstrate remarkable proficiency in simulating physical laws, they frequently fail to account for the "irrational" nuances of human behavior driven by internal affective states. This paper introduces the Large Emotional World Model (LEWM), a framework designed to bridge this "empathy gap." By explicitly modeling emotional transitions alongside physical states, LEWM provides a more comprehensive understanding of human social dynamics. Central to this work is the Emotion-Why-How (EWH) dataset, a multimodal benchmark that maps behavioral transitions through the lens of emotional causality. Our results indicate that integrating affective circuits enhances subjective reasoning without significantly compromising performance on objective logical benchmarks.

The Empathy Gap in AI World Models

Most existing AI world models function primarily as physics simulators. They excel at predicting the next frame of a video or the trajectory of an object based on gravity, collisions, and spatial logic. However, humans are not merely physical entities; we are emotional agents whose actions are often dictated by feelings rather than pure utility.

Comparison between a standard World Model and an Emotional World Model in predicting human behavior.

A standard physics-based or "rational" model might struggle to explain why a person feeling sad might engage in impulsive shopping—an action that is financially suboptimal but emotionally restorative. This limitation hinders AI performance in social simulations, human-robot interaction, and autonomous driving, where understanding human intent is paramount. The LEWM addresses this by seeking to understand the affective "Why" behind the behavioral "How."

Methodology: Integrating Emotion into World Modeling

The researchers developed a two-stage approach to validate and implement emotional intelligence within a world model framework.

1. Emotion-Filtering Validation

To establish the necessity of emotional context, the team first built a module to "neutralize" emotional text. By comparing a standard model to one where emotional nuances were stripped away, they observed a significant decline in performance on subjective reasoning tasks, proving that affect is a functional component of human-like reasoning.

2. LEWM Architecture

The core of the LEWM is a multimodal architecture designed to process video, audio, and facial expressions simultaneously.

The LEWM architecture utilizes multimodal encoders and a transformer decoder to predict future states conditioned on emotional transitions.
  • Multimodal Encoders: The system employs specialized encoders for the visual state ($S_t$), the emotional state ($E_t$, derived from facial features and context), and actions ($A_t$).
  • Coupled Transition Factorization: This "emotion-first" causal pathway is the model's primary innovation. Instead of predicting the next environment state directly, the model first predicts the evolution of the internal emotional state ($E_{t+1}$). This updated emotional context then conditions the prediction of the subsequent environmental state ($S_{t+1}$).
  • Emotion-Consistency Regularization: To prevent jittery or unrealistic behavioral predictions, the model uses a consistency loss function. This ensures that predicted world changes remain stable under minor emotional fluctuations.

The Emotion-Why-How (EWH) Dataset

To train and evaluate the LEWM, the authors introduced the Emotion-Why-How (EWH) dataset. This multimodal benchmark consists of sequences from movies, television shows, and egocentric recordings where human behavior is explicitly driven by emotion. The dataset maps the complex transition: $(State_0, Emotion_0) \to Action \to Emotion_1 \to State_1$.

In addition to EWH, the model was validated using the MELD dataset for multimodal sentiment recognition, and standard benchmarks like HellaSwag and MMLU to ensure general reasoning capabilities remained intact.

Key Results and Insights

The study yields several critical insights into the role of emotion in artificial intelligence:

  • Emotion as a Systematic Modulator: Emotion is not merely "noise"; it acts as a systematic modulator of behavior. Filtering out emotional signals resulted in an 8% drop in accuracy on subjective tasks.
  • Theory of Mind: LEWM is inspired by the psychological concept of "Theory of Mind"—the ability to attribute mental states to others. By explicitly modeling internal affect ($E_t$), the AI achieves a more human-like understanding of social causality.
  • Performance Stability: Crucially, the addition of emotional intelligence did not "break" the model's logical faculties. LEWM maintained comparable performance to general world models on objective benchmarks, with only a 1% drop on MMLU and a 3% drop on HellaSwag.
  • Predictive Accuracy: In sentiment and emotion tasks using the MELD dataset, removing emotional cues led to a 10% drop in F1-score, highlighting the necessity of affective circuits for accurate social prediction.

Limitations and Future Work

While the LEWM represents a significant step forward, several challenges remain. The emotional labels in the EWH dataset were generated via weak supervision from other Large Multimodal Models (LMMs), which may introduce noise compared to human-expert labeling. Furthermore, real-world social interactions are vastly more complex than the cinematic clips currently captured in the dataset.

Future research will focus on scaling the EWH dataset and refining "affective circuits" to handle more nuanced, long-term emotional transitions, such as the cumulative effect of stress on decision-making over time.

Summary updated

Abstract

The world is governed by both physical laws and affective dynamics. Physical laws govern state transitions, while affective dynamics shape human actions, decisions, and interactions. A world model that learns only physical laws can approximate the physical world, but not the human world. In this paper, we introduce human emotion as a key state variable in world models, enabling them to capture both future state transitions and their emotional causes. We first construct Emotion-Why-How (EWH), the first world model dataset centered on emotional state transitions, containing 10,850 emotion-aware transition tuples. Each tuple encodes the pre-state, pre-emotion, action, post-emotion, and post-state, supporting reasoning about why actions occur and how emotions reshape future states. Based on EWH, we propose the Large Emotional World Model (LEWM), which factorizes future prediction into two coupled steps: first predicting the future emotional state from the current context, and then conditioning future world-state prediction on the predicted emotion. Experiments show that LEWM brings consistent gains across world-state prediction, emotion understanding, and general reasoning tasks. It achieves up to 45.72% accuracy improvement on EWH, 3.94% on WorldNet, 17.47% F1 improvement on MELD, and a 6.10% gain on specific MMLU categories. These results demonstrate that incorporating emotion into world models enables more realistic simulation of human-centered environments and expands the predictive understanding of intelligent agents.
BibTeX
@misc{zhang2025large,
  author = {Yazhou Zhang and Changhao Song and Yazhou Zhang and Hui Gao and Chang Yang and Peng Zhang},
  title = {{Large Emotional World Model}},
  year = {2025},
  eprint = {2512.24149},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.24149},
}
Figure