Beyond Winning and Losing: Modeling Human Motivations and Behaviors with Vector-Valued Inverse Reinforcement Learning

Baoxiang Wang, Tongfang Sun, Xianjun Sam ZhengAIIDE 2019 · OpenReview

TL;DR

This paper introduces a novel approach to Inverse Reinforcement Learning (IRL) that models the multifaceted motivations of human players in games, moving beyond simply optimizing for winning. By extending IRL to handle vector-valued rewards and employing Pareto optimality, the method significantly relaxes prior assumptions, leading to more realistic behavior modeling and improved prediction accuracy on complex game environments.

What Problem Does This Paper Address?

Traditional Reinforcement Learning (RL) and Inverse Reinforcement Learning (IRL) methods have achieved remarkable success in game AI, often surpassing human performance. However, these successes are typically measured by a single objective: winning the game or achieving a high score. This narrow focus overlooks the rich tapestry of human motivations that drive gameplay, such as enjoyment, social interaction, exploration, and escapism. Understanding these diverse motivations is crucial for accurately modeling human behavior in complex environments. Existing IRL methods often rely on strong assumptions about agent optimality and scalar reward functions, which are insufficient for capturing the nuanced, multi-objective nature of human decision-making in games. This paper aims to bridge this gap by developing an IRL framework that explicitly models multiple, potentially conflicting, human motivations.

Methods

The core contribution of this paper is the development of a Multi-Motivation Behavior Modeling (MMBM) framework. This approach extends Inverse Reinforcement Learning (IRL) to handle vector-valued rewards, where each dimension of the reward vector represents a different motivation (e.g., achievement, social, immersion).

Key methodological aspects include:

  • Vector-Valued MDP: The problem is framed within a Vector-Valued Markov Decision Process (MDP), where the agent's utility is a vector, and optimality is defined using Pareto dominance rather than a single scalar value. This means an action is optimal if no other action yields a strictly better outcome in at least one motivation dimension without sacrificing performance in others.
  • Relaxed IRL Assumption: MMBM significantly weakens the traditional IRL assumption of strict optimality. Instead, it assumes that an observed action is either not dominated by any alternative feasible action or yields an identical utility vector. This is crucial for modeling real-world behaviors where players might not always act optimally according to a single objective.
  • Value Function Approximation: The paper utilizes a Q-learning-based approach to approximate the action-state value functions for each motivation dimension. This involves learning d separate Q-functions, where d is the number of motivations.
  • Estimating the Value Function Range: A key theoretical contribution is the characterization of the set of optimal policies as a convex hull of extreme points. The paper proves that the range of the value function for a vector-valued MDP can be approximated by the convex hull of the value functions corresponding to policies optimized for individual reward components (e.g., maximizing only achievement).
  • Distance Minimization: The framework measures the suboptimality of demonstrated behavior by computing the distance between the observed behavior's value and the estimated range of optimal values. This distance is minimized using a linear program, which also helps in estimating the projection vector y that represents the relative importance of each motivation.

Datasets

The proposed MMBM algorithm was tested on the World of Warcraft Avatar History (WoWAH) dataset. This dataset contains gameplay data, including player movements (actions) and observations (states), collected over a three-year period from the TW-Light's Hope realm. The authors treat each player as an agent and their trajectories as demonstrations of behavior.

Key Insights and Ideas

  • Beyond Winning: The most significant insight is that human behavior in games is driven by a complex interplay of motivations, not just the desire to win. MMBM's ability to model these multiple motivations leads to a more nuanced understanding of player behavior.
  • Pareto Optimality for Realism: By adopting Pareto optimality, the framework allows for a more realistic representation of decision-making, acknowledging that players might make choices that are optimal across several objectives, even if not strictly optimal for a single one. This is a substantial departure from traditional IRL assumptions.
  • Weakening IRL Assumptions: The relaxed optimality assumption is critical. It allows the model to account for suboptimal actions or exploration, which are common in human gameplay, leading to more robust and accurate behavior modeling.
  • Understanding Player Archetypes: The experiments reveal distinct motivational profiles for different player groups (e.g., high-level vs. low-level players, different classes, guild members), providing valuable insights into player psychology and game design.

Key Results

  • Improved Prediction Accuracy: MMBM achieved significantly higher accuracy in predicting player actions compared to existing IRL methods, including linear scalarization, max-entropy scalarization, and large-margin scalarization. For instance, the proposed approach with c=0.5 achieved 85.5% accuracy, compared to around 48-59% for other methods.
  • Reduced Inverse Learning Error (ILE): The model also demonstrated a lower inverse learning error (13.2% for c=0.5), indicating a more accurate estimation of the underlying reward functions and player motivations.
  • Identification of Motivational Differences: The spider maps in Figure 2 clearly illustrate how player motivations differ based on game level, class, and guild membership, aligning with existing qualitative research on World of Warcraft.
  • Trend Analysis: Analysis of motivation trends over time (Figure 3) showed shifts in player focus, particularly an increase in advancement and competition motivations coinciding with game updates like the "Wrath of the Lich King" expansion.

Limitations and Future Work

  • Sensitivity to Reward Function Accuracy: The authors acknowledge that their algorithm's performance can be sensitive to the accuracy of the reward function components. If one component is inaccurate, the model might incur larger errors than methods that scalarize the reward, as it cannot easily adjust weights to compensate.
  • Future Work: Robust Reward Learning: To address the sensitivity issue, future work could focus on learning a robust function for each reward element, potentially moving beyond simple linear function approximation to a more subtle functional space. This would allow the algorithm to better handle inaccuracies and learn more effectively from complex, multi-objective reward structures.
  • Generalization to Other Environments: While tested on World of Warcraft, the framework's applicability to other complex, interactive environments is a promising avenue for future research.

Summary updated

Abstract

In recent years, reinforcement learning (RL) methods have been applied to model gameplay with great success, achieving super-human performance in various environments, such as Atari, Go and Poker. However, these studies mostly focus on winning the game and have largely ignored the rich and complex human motivations, which are essential for understanding humans’ diverse behavior. In this paper, we present a multi-motivation behavior model which investigates the multifaceted human motivations and learns the underlying value structure of the agents. Our approach extends inverse RL to vectored-valued rewards with Pareto optimality which significantly weakens the inverse RL assumption. Our model therefore incorporates a wider range of behavior that commonly appears in real-world environments. For practical assessment, our algorithm is tested on World of Warcraft datasets and demonstrates the improvement over existing methods.
BibTeX
@inproceedings{wang2019beyond,
  author = {Baoxiang Wang and Tongfang Sun and Xianjun Sam Zheng},
  title = {{Beyond Winning and Losing: Modeling Human Motivations and Behaviors with Vector-Valued Inverse Reinforcement Learning}},
  booktitle = {AIIDE},
  year = {2019},
  url = {https://openreview.net/forum?id=alxLr_M4CCn},
}
Figure