Introduction
In classical economic modeling, "persuasion games" are used to study interactions between an informed sender (the expert) and a decision-maker (the receiver). Traditionally, these models rely on abstract or numerical signals—such as a simple quality score—to represent information. However, in the real world, persuasion is mediated through natural language: sales pitches, online reviews, and advertisements.
Natural language introduces complexities that numerical signals lack, including vagueness, sentiment, and the capacity for strategic exaggeration. This research bridges the gap between game theory and natural language processing (NLP) by investigating how humans make decisions when presented with textual evidence and how well machine learning models can predict those choices.
Experimental Framework
The researchers designed a controlled, 10-round online experiment involving 509 pairs of participants recruited via Amazon Mechanical Turk. The participants were assigned roles in a "Hotel Booking" game:
- The Expert: Provided with seven hotel reviews from a Booking.com dataset, each with an associated numerical score. The Expert’s goal is to convince the Decision-Maker to book the hotel by selecting and presenting exactly one of these reviews.
- The Decision-Maker (DM): Receives the selected review and must choose between two actions: "Book the Hotel" (a risky choice with potential high payoff) or "Stay Home" (a safe choice with a guaranteed but lower payoff).
This setup allowed the researchers to collect a dataset of 5,090 individual interactions, providing a rich basis for modeling human behavior in a strategic, language-based environment.
Modeling Strategies
The study compared several computational approaches to predict the Decision-Maker's choices based on the text they received and the history of the game:
- Linear Classifiers (SVM): Used as a non-sequential baseline to predict decisions based on the current round's text.
- Sequential Models (LSTM): Specifically the LSTM-TRCR architecture, which captures how trust and reputation evolve over the 10-round sequence by jointly predicting per-trial decisions and the overall choice rate.
- Attention-based Models (Transformers): Treating the game history as a sequence-to-sequence task.
Crucially, the researchers compared two types of text representations:
- BERT-based Embeddings: High-dimensional vectors from a pre-trained Transformer model.
- Hand-Crafted Features (HCF): A set of 42 binary features identifying specific topics (e.g., mentions of the pool), sentiment intensity, and structural cues (e.g., "explicitly states there is nothing negative").
Key Insights and Results
1. The Superiority of Hand-Crafted Features
Surprisingly, models using simple hand-crafted features significantly outperformed deep learning models like BERT. While BERT is a powerhouse for general NLP tasks, it struggled to predict human decisions in this context, often defaulting to predicting the majority class (always booking). The HCF-based models achieved a per-trial accuracy of approximately 77% and were far more effective at identifying the "minority class"—the instances where a DM chooses to "Stay Home."
2. Words vs. Numbers
The study found that when the game used verbal reviews instead of raw numerical scores, player behavior aligned more closely with game-theoretic equilibrium. Natural language appears to provide a "softer," more calibrated signal that humans are better at interpreting than raw statistics.
3. Strategic Dynamics and Trust
As the 10-round game progressed, Experts learned to become more strategic, frequently choosing reviews with "implied scores" much higher than the hotel's actual average. In response, Decision-Makers' trust tended to decline over time. One of the most persuasive linguistic features identified was not a glowing superlative, but the phrase "nothing negative," which served as a powerful signal of reliability to the DMs.
4. Interpretability
Using SHAP (SHapley Additive exPlanations), the authors demonstrated that specific semantic concepts were highly predictive of refusal. For example, mentions of the room in a negative context were strong indicators that a DM would choose to stay home, regardless of other positive attributes mentioned in the review.
Conclusion and Future Work
The findings suggest that the features BERT learns for general language modeling are not necessarily the same features humans prioritize when evaluating trust and making strategic decisions. Hand-crafted features, though domain-specific, capture the psychological nuances of persuasion more effectively.
Future research aims to move from predicting decisions to generating persuasive text. By understanding the linguistic markers that drive human choice, AI agents could be developed to communicate more effectively in strategic, game-theoretic settings, potentially moving toward a more universal set of persuasive linguistic features across different domains.
Summary updated