Predicting Decisions in Language Based Persuasion Games

Reut Apel, Ido Erev, Roi Reichart, Moshe Tennenholtz

Introduction

In classical economic modeling, "persuasion games" are used to study interactions between an informed sender (the expert) and a decision-maker (the receiver). Traditionally, these models rely on abstract or numerical signals—such as a simple quality score—to represent information. However, in the real world, persuasion is mediated through natural language: sales pitches, online reviews, and advertisements.

Natural language introduces complexities that numerical signals lack, including vagueness, sentiment, and the capacity for strategic exaggeration. This research bridges the gap between game theory and natural language processing (NLP) by investigating how humans make decisions when presented with textual evidence and how well machine learning models can predict those choices.

Experimental Framework

The researchers designed a controlled, 10-round online experiment involving 509 pairs of participants recruited via Amazon Mechanical Turk. The participants were assigned roles in a "Hotel Booking" game:

  • The Expert: Provided with seven hotel reviews from a Booking.com dataset, each with an associated numerical score. The Expert’s goal is to convince the Decision-Maker to book the hotel by selecting and presenting exactly one of these reviews.
  • The Decision-Maker (DM): Receives the selected review and must choose between two actions: "Book the Hotel" (a risky choice with potential high payoff) or "Stay Home" (a safe choice with a guaranteed but lower payoff).

This setup allowed the researchers to collect a dataset of 5,090 individual interactions, providing a rich basis for modeling human behavior in a strategic, language-based environment.

Modeling Strategies

The study compared several computational approaches to predict the Decision-Maker's choices based on the text they received and the history of the game:

  1. Linear Classifiers (SVM): Used as a non-sequential baseline to predict decisions based on the current round's text.
  2. Sequential Models (LSTM): Specifically the LSTM-TRCR architecture, which captures how trust and reputation evolve over the 10-round sequence by jointly predicting per-trial decisions and the overall choice rate.
  3. Attention-based Models (Transformers): Treating the game history as a sequence-to-sequence task.

Crucially, the researchers compared two types of text representations:

  • BERT-based Embeddings: High-dimensional vectors from a pre-trained Transformer model.
  • Hand-Crafted Features (HCF): A set of 42 binary features identifying specific topics (e.g., mentions of the pool), sentiment intensity, and structural cues (e.g., "explicitly states there is nothing negative").

Key Insights and Results

1. The Superiority of Hand-Crafted Features

Surprisingly, models using simple hand-crafted features significantly outperformed deep learning models like BERT. While BERT is a powerhouse for general NLP tasks, it struggled to predict human decisions in this context, often defaulting to predicting the majority class (always booking). The HCF-based models achieved a per-trial accuracy of approximately 77% and were far more effective at identifying the "minority class"—the instances where a DM chooses to "Stay Home."

2. Words vs. Numbers

The study found that when the game used verbal reviews instead of raw numerical scores, player behavior aligned more closely with game-theoretic equilibrium. Natural language appears to provide a "softer," more calibrated signal that humans are better at interpreting than raw statistics.

3. Strategic Dynamics and Trust

As the 10-round game progressed, Experts learned to become more strategic, frequently choosing reviews with "implied scores" much higher than the hotel's actual average. In response, Decision-Makers' trust tended to decline over time. One of the most persuasive linguistic features identified was not a glowing superlative, but the phrase "nothing negative," which served as a powerful signal of reliability to the DMs.

4. Interpretability

Using SHAP (SHapley Additive exPlanations), the authors demonstrated that specific semantic concepts were highly predictive of refusal. For example, mentions of the room in a negative context were strong indicators that a DM would choose to stay home, regardless of other positive attributes mentioned in the review.

Conclusion and Future Work

The findings suggest that the features BERT learns for general language modeling are not necessarily the same features humans prioritize when evaluating trust and making strategic decisions. Hand-crafted features, though domain-specific, capture the psychological nuances of persuasion more effectively.

Future research aims to move from predicting decisions to generating persuasive text. By understanding the linguistic markers that drive human choice, AI agents could be developed to communicate more effectively in strategic, game-theoretic settings, potentially moving toward a more universal set of persuasive linguistic features across different domains.

Summary updated

Abstract

Sender-receiver interactions, and specifically persuasion games, are widely researched in economic modeling and artificial intelligence. However, in the classic persuasion games setting, the messages sent from the expert to the decision-maker (DM) are abstract or well-structured signals rather than natural language messages. This paper addresses the use of natural language in persuasion games. For this purpose, we conduct an online repeated interaction experiment. At each trial of the interaction, an informed expert aims to sell an uninformed decision-maker a vacation in a hotel, by sending her a review that describes the hotel. While the expert is exposed to several scored reviews, the decision-maker observes only the single review sent by the expert, and her payoff in case she chooses to take the hotel is a random draw from the review score distribution available to the expert only. We also compare the behavioral patterns in this experiment to the equivalent patterns in similar experiments where the communication is based on the numerical values of the reviews rather than the reviews' text, and observe substantial differences which can be explained through an equilibrium analysis of the game. We consider a number of modeling approaches for our verbal communication setup, differing from each other in the model type (deep neural network vs. linear classifier), the type of features used by the model (textual, behavioral or both) and the source of the textual features (DNN-based vs. hand-crafted). Our results demonstrate that given a prefix of the interaction sequence, our models can predict the future decisions of the decision-maker, particularly when a sequential modeling approach and hand-crafted textual features are applied. Further analysis of the hand-crafted textual features allows us to make initial observations about the aspects of text that drive decision making in our setup
BibTeX
@article{apel2022predicting,
  author = {Reut Apel and Ido Erev and Roi Reichart and Moshe Tennenholtz},
  title = {{Predicting Decisions in Language Based Persuasion Games}},
  journal = {J. Artif. Intell. Res.},
  year = {2022},
  doi = {10.1613/jair.1.13510},
}

Loading related content...

Figure