Predicting Decisions in Language Based Persuasion Games

Reut Apel, Ido Erev, Roi Reichart, Moshe Tennenholtz

Abstract

Sender-receiver interactions, particularly persuasion games, are fundamental to economic modeling and artificial intelligence. Traditionally, these games rely on abstract or well-structured signals. This paper investigates the introduction of natural language into persuasion games through an online repeated interaction experiment. In this setup, an informed expert attempts to persuade an uninformed decision-maker (DM) to book a hotel by selecting specific reviews. Our study compares behavioral patterns between verbal and numerical communication, finding that natural language leads to more predictable and rational behavior. We evaluate several modeling approaches, including deep neural networks and linear classifiers, using both BERT-based and hand-crafted textual features. Our results indicate that sequential modeling combined with hand-crafted features achieves high accuracy in predicting DM decisions, offering new insights into the linguistic drivers of strategic decision-making.

Introduction

In traditional game theory, "persuasion games" involve an informed expert attempting to influence the actions of an uninformed decision-maker. Historically, these models have relied on numeric signals or structured data. However, real-world persuasion—from marketing to interpersonal negotiations—is conducted through natural language. This research addresses the gap between abstract economic models and the complexity of human language by analyzing how verbal communication influences strategic choices in a multi-round simulation.

Experimental Design: The Hotel Booking Game

To study these dynamics, we conducted an online experiment via Amazon Mechanical Turk involving 408 pairs of participants. The interaction was structured as a 10-round game between an "Expert" (acting as a travel agent) and a "Decision-Maker" (the client).

In each round:

  1. The Expert is presented with seven reviews and associated scores for a specific hotel, sourced from a dataset of 500,000 Booking.com reviews.
  2. The Expert selects exactly one review to show the Decision-Maker.
  3. The Decision-Maker observes only that single review and must decide whether to "Book" or "Stay Home."
  4. The Payoff: If the DM books, their payoff is a random draw from the hotel's actual score distribution (known only to the Expert). If they stay home, they receive a neutral default payoff.

This setup creates a strategic environment where the Expert must decide how much information to reveal to maximize bookings, while the DM must learn to interpret the Expert's signals over time.

Modeling and Feature Engineering

We compared several computational architectures to predict the Decision-Maker’s choices based on the history of the interaction:

  • Support Vector Machines (SVM): Used as a non-sequential baseline to evaluate individual trials in isolation.
  • Long Short-Term Memory (LSTM) Networks: Employed to capture the sequential nature of the 10-round game, accounting for how previous rounds influence current trust.
  • Transformers: A sequence-to-sequence approach treating the game history as a translation-style task.

We also contrasted two types of textual representations:

  1. BERT-based Embeddings: Standard deep learning representations that capture broad linguistic context.
  2. Hand-Crafted Features (HCF): A set of 42 binary features identifying specific topics (e.g., mentions of the room or staff), sentiment intensity, and structural elements (e.g., the presence of a positive summary sentence).

Key Findings

1. Language Increases "Rationality"

One of the most striking observations was that verbal communication made human behavior more predictable and closer to a game-theoretic "pooling equilibrium" than numeric signals alone. In numeric versions of this game, players often struggle to find a stable strategy. In the verbal version, the nuances of language provided a more intuitive framework for establishing trust and strategic consistency.

2. The Superiority of Hand-Crafted Features

Contrary to the current trend favoring end-to-end deep learning, our Hand-Crafted Features (HCF) outperformed raw BERT embeddings. This suggests that in strategic contexts, high-level semantic concepts—such as specifically mentioning that there were "no negatives"—carry more weight than the general-purpose patterns learned by large language models. The best-performing model, a joint LSTM using HCF, achieved a per-trial prediction accuracy of 76.9%.

3. Sequentiality and Trust

Decision-makers do not evaluate reviews in a vacuum. Our models showed that sequentiality is key: a DM’s decision in round five is heavily influenced by whether the Expert provided an accurate or misleading review in rounds one through four. LSTMs significantly outperformed non-sequential models by accounting for this "memory" of the interaction.

4. Linguistic Nudges

Using SHAP (interpretability analysis), we identified the specific linguistic features that most strongly drove decisions. The most impactful features included:

  • Explicit "nothing negative" statements.
  • High-intensity positive descriptors.
  • Mentions of the "room" within a negative context (which served as a strong deterrent).

Conclusion and Future Work

This research demonstrates that natural language is not just a medium for information, but a strategic tool that shapes the rationality of human agents in persuasion games. By combining sequential modeling with targeted linguistic features, we can accurately predict how humans will respond to persuasive text.

Future research will look toward moving from prediction to generation—developing AI agents capable of generating the most persuasive possible text based on the evolving state of a strategic interaction. Additionally, moving beyond the controlled environment of Mechanical Turk to high-stakes, real-world decision-making remains a vital next step.

Summary updated

Abstract

Sender-receiver interactions, and specifically persuasion games, are widely researched in economic modeling and artificial intelligence. However, in the classic persuasion games setting, the messages sent from the expert to the decision-maker (DM) are abstract or well-structured signals rather than natural language messages. This paper addresses the use of natural language in persuasion games. For this purpose, we conduct an online repeated interaction experiment. At each trial of the interaction, an informed expert aims to sell an uninformed decision-maker a vacation in a hotel, by sending her a review that describes the hotel. While the expert is exposed to several scored reviews, the decision-maker observes only the single review sent by the expert, and her payoff in case she chooses to take the hotel is a random draw from the review score distribution available to the expert only. We also compare the behavioral patterns in this experiment to the equivalent patterns in similar experiments where the communication is based on the numerical values of the reviews rather than the reviews' text, and observe substantial differences which can be explained through an equilibrium analysis of the game. We consider a number of modeling approaches for our verbal communication setup, differing from each other in the model type (deep neural network vs. linear classifier), the type of features used by the model (textual, behavioral or both) and the source of the textual features (DNN-based vs. hand-crafted). Our results demonstrate that given a prefix of the interaction sequence, our models can predict the future decisions of the decision-maker, particularly when a sequential modeling approach and hand-crafted textual features are applied. Further analysis of the hand-crafted textual features allows us to make initial observations about the aspects of text that drive decision making in our setup
BibTeX
@article{apel2022predicting,
  author = {Reut Apel and Ido Erev and Roi Reichart and Moshe Tennenholtz},
  title = {{Predicting Decisions in Language Based Persuasion Games}},
  journal = {J. Artif. Intell. Res.},
  year = {2022},
  doi = {10.1613/jair.1.13510},
}

Loading related content...

Figure