In the evolving landscape of human-AI interaction, we often ask: can an AI truly understand us? A new paper, "Words or Deeds? How LLMs Predict Users’ Decisions in Human–Agent Dialogue," suggests that while LLMs are getting better at following the conversation, they are surprisingly gullible when it comes to predicting what we’ll actually do.
TL;DR
This study reveals that LLMs over-rely on a user's stated intentions ("words") while ignoring their past behavior ("deeds") and the social context of the interaction. In mixed-motive games, this leads LLMs to be easily "tricked" by broken promises, failing to capture the nuanced ways human behavior shifts depending on who they are talking to.
What Problem Does This Paper Address?
As LLMs are deployed as sales, service, or negotiation agents, they must predict user decisions to act effectively. However, in "mixed-motive" scenarios—where interests partly align and partly conflict—users often engage in "cheap talk." They might promise to cooperate to gain a concession and then immediately defect.
The core challenge is grounding: should a system believe what a user says, or what they have done in the past? If a predictor relies too heavily on promises, it becomes vulnerable to exploitation. Conversely, ignoring dialogue might miss genuine changes of heart. This paper investigates how current open-weight LLMs navigate this "words vs. deeds" trade-off.
Methods
The researchers used a repeated prisoner’s dilemma-style game to create a controlled environment for "cheap talk." 1. The Players: 42 human participants played against five different AI agents. 2. The Personas: Each agent was a GPT-4o model prompted to exhibit one of the "Big Five" personality traits (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism). 3. The Predictors: Eight open-weight LLMs (including Qwen, Gemma, and GLM families) acted as "offline observers." They were given various combinations of the decision record (deeds) and the dialogue (words) and asked to predict the user's next move.
The team performed an extensive ablation study, testing 19 different information conditions to see exactly which pieces of data (past games, current game history, current round dialogue) most influenced the LLMs' predictions.
Datasets
The study utilized the Personality-1260 dataset, which contains 1,260 rounds of human-agent interaction. This dataset is unique because it links a user’s personality scores, their conversational statements, and their actual subsequent actions across multiple rounds and different agent personas.
Key Insights and Ideas
- Dialogue is a Distractor: Surprisingly, adding dialogue to the decision record provided no detectable gain in overall prediction ranking (AUC). The LLMs were already extracting most of the useful predictive signal from the user's past actions.
- The "Gullibility" Gap: LLMs are far more optimistic about human honesty than reality warrants. When a user made a promise they hadn't made before, LLMs boosted the predicted probability of cooperation from 13% to 69%. In reality, humans only kept those specific promises 40% of the time.
- Context Blindness: Humans are social chameleons; we cooperate more with "Agreeable" agents (70.6%) than "Extraverted" ones (54.5%). However, the LLM predictors were almost entirely blind to this, predicting nearly identical cooperation rates regardless of the agent's persona.
- Averages Conceal Failures: On the surface, adding dialogue seemed to have little effect on accuracy. But looking deeper, the researchers found that dialogue raised accuracy on kept promises while crashing it on broken ones. These two effects canceled each other out in aggregate metrics, hiding a significant failure mode.
Key Results
- Baseline Performance: Predictions based on the decision record alone achieved an AUC of 0.80, showing that "deeds" are a strong predictor of future behavior.
- The Cost of Words: For rounds where users broke their word, adding dialogue caused prediction accuracy to drop from 0.65 to a dismal 0.17.
- Persona Bias: While users' actual cooperation varied by over 16 percentage points between agent personas, LLM predictions varied by only 1.4 points.
- Trait Compatibility: Users cooperated more when their own personality matched the agent's (e.g., an agreeable user with an Agreeable agent). The LLMs failed to predict this "compatibility" effect, even when explicitly given the user's personality scores in the prompt.
Limitations and Future Work
The authors acknowledge several limitations:
- Population: The study was limited to 42 Chinese university students in a low-stakes environment.
- Zero-Shot Constraints: The predictors were used in a zero-shot capacity with reasoning disabled. Chain-of-thought or fine-tuning might alter these results.
- Causality: The study identifies associations but cannot definitively say that the agent's language caused the change in user behavior (as opposed to the agent's decision policy).
Future work should explore "history-aware" or "counterpart-conditioned" predictors that explicitly weigh a user's current promise against their historical "word-keeping" rate. As the authors conclude, for an AI to be a truly effective partner, it must learn to trust, but verify.
Summary updated