TL;DR
This paper introduces $M^3IRL$, a novel Inverse Reinforcement Learning framework designed to uncover the hidden motivations (cost functions) of actors in social networks. Unlike traditional methods, it explicitly accounts for non-Markovian human memory and asymmetric cognitive biases—like confirmation and novelty bias—requiring only a single trajectory of observed behavior to make accurate inferences.
What Problem Does This Paper Address?
In the fight against misinformation and digital polarization, game-theoretic models are powerful tools. However, these models typically assume we already know the "cost functions" (the underlying goals and priorities) of the players involved. Learning these functions from observation is a classic problem known as Inverse Reinforcement Learning (IRL).
Standard IRL approaches, often used in robotics, face two major hurdles in social contexts:
- Human Memory: Most IRL methods rely on the Markov Decision Process (MDP) principle, where the next state depends only on the current one. Humans, however, have memories. Our current opinions are shaped by a "memory horizon" of past information, breaking the MDP assumption.
- Cognitive Asymmetry: Human bias isn't symmetric. For example, George might value Alex’s opinion more than Bob’s, even if both are equally "distant" from his own, simply because Alex’s view aligns better with his existing beliefs (confirmation bias).
The authors set out to build a model that captures these human quirks and an algorithm that can infer motivations from limited, real-world social media data.
Methods
The authors propose a two-part solution:
1. A "Memorized" Social Diffusion Model
They develop a dynamic model (Eq. 1) where an individual's opinion at time $k+1$ is a function of their subconscious bias, their own memory, and the influence of their neighbors. Crucially, the influence weights are state-dependent and capture:
- Confirmation Bias: Giving more weight to information that confirms existing beliefs.
- Novelty Bias: A preference for outlying or "surprising" content.
- Memory Horizons: Using a decaying influence function (inspired by the ACT-R cognitive architecture) to model how past information fades over time.
2. The $M^3IRL$ Algorithm
Standing for Memorized Model and Maximum-entropy based Inverse Reinforcement Learning, this algorithm avoids the need for massive datasets.
- Local Optimality: Instead of assuming players are globally optimal, it assumes they are locally optimal, which is more realistic for human decision-making.
- Single Trajectory: It can infer cost functions from just one finite-time sequence of opinions.
- Bounded Variables: It handles the fact that opinions are typically bounded (e.g., between -1 for "completely supporting" and +1 for "completely opposing").
Datasets
The researchers validated their model using real-world data:
- Twitter Vaccine Data: They collected tweets from nine users over nine months (Jan–Sept 2021) regarding COVID-19 vaccines.
- Sentiment Encoding: Tweets were processed using a BERT sentiment analysis model trained on the Sentiment140 dataset, mapping opinions to a numerical range of [-1, 1].
- Network Structure: They mapped the interaction digraph of these nine users, identifying seven "humans" and two "information sources" (targets for cost function learning).
Key Insights and Ideas
- Asymmetry is Key: The paper formalizes why "distance" in opinion space isn't enough. As shown in their "Alex, Bob, and George" scenarios, the direction of the distance relative to one's own belief domain drastically changes influence weights.
- Breaking the MDP: By incorporating a memory horizon ($\tau_i$), the model moves beyond the "memoryless" limitations of standard reinforcement learning, making it far better suited for modeling human social behavior.
- Dynamic Cost Functions: The authors demonstrate that cost functions aren't static; they evolve based on external stimuli (like news events), and $M^3IRL$ is sensitive enough to capture these shifts using recent trajectory data.
Key Results
The algorithm successfully identified the distinct motivations of the two observed information sources (IS8 and IS9):
- IS8 (Anti-Vaccine): The learned cost function for IS8 showed a strong preference for steering public opinion toward the "opposing" end of the spectrum (-1).
- IS9 (Pro-Vaccine): Conversely, IS9's cost function was heavily weighted toward steering the public toward "supporting" the vaccine (+1).
- Influence Differential: The analysis revealed that IS9 had a significantly higher influence ($\alpha_9 = 0.7960$) on the community's evolving opinions compared to IS8 ($\alpha_8 = 0.2040$).
- Behavioral Traits: Both sources showed a preference for spreading "outlying" opinions rather than persistent, unchanging ones, which was reflected in the sharp jumps observed in their opinion trajectories.
Limitations and Future Work
- Scale: The empirical validation was performed on a small network of nine users. The authors acknowledge that scaling this to massive social networks is a significant future challenge.
- Basis Function Library: The method relies on a predefined library of basis functions (e.g., steering toward +1, stubbornness). If the true motivation of a player isn't represented in the library, the inference may be inaccurate.
- Future Directions: The team plans to investigate how to incorporate communication detection and classification to generalize the framework for large-scale social network analysis.
Summary updated