Words or Deeds? How LLMs Predict Users' Decisions in Human-Agent Dialogue

Baiqiao Zhang, Shuai Ma, Zhifeng Liao, Chao Zhou, Jianshu Wang, Yuhan Zhao, Chaoyue Zhang, Tongjun Si, Yuying Tang, Xiangxian Li, Yong-Jin Liu, Yulong BianarXiv 2026 · arXiv

In the evolving landscape of human-AI interaction, we often ask: can an AI truly understand us? A new paper, "Words or Deeds? How LLMs Predict Users’ Decisions in Human–Agent Dialogue," suggests that while LLMs are getting better at following the conversation, they are surprisingly gullible when it comes to predicting what we’ll actually do.

TL;DR

This study reveals that LLMs over-rely on a user's stated intentions ("words") while ignoring their past behavior ("deeds") and the social context of the interaction. In mixed-motive games, this leads LLMs to be easily "tricked" by broken promises, failing to capture the nuanced ways human behavior shifts depending on who they are talking to.

What Problem Does This Paper Address?

As LLMs are deployed as sales, service, or negotiation agents, they must predict user decisions to act effectively. However, in "mixed-motive" scenarios—where interests partly align and partly conflict—users often engage in "cheap talk." They might promise to cooperate to gain a concession and then immediately defect.

The core challenge is grounding: should a system believe what a user says, or what they have done in the past? If a predictor relies too heavily on promises, it becomes vulnerable to exploitation. Conversely, ignoring dialogue might miss genuine changes of heart. This paper investigates how current open-weight LLMs navigate this "words vs. deeds" trade-off.

Methods

The researchers used a repeated prisoner’s dilemma-style game to create a controlled environment for "cheap talk." 1. The Players: 42 human participants played against five different AI agents. 2. The Personas: Each agent was a GPT-4o model prompted to exhibit one of the "Big Five" personality traits (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism). 3. The Predictors: Eight open-weight LLMs (including Qwen, Gemma, and GLM families) acted as "offline observers." They were given various combinations of the decision record (deeds) and the dialogue (words) and asked to predict the user's next move.

The experimental design showing how LLMs predict decisions from deeds (past rounds) and words (current dialogue).

The team performed an extensive ablation study, testing 19 different information conditions to see exactly which pieces of data (past games, current game history, current round dialogue) most influenced the LLMs' predictions.

Datasets

The study utilized the Personality-1260 dataset, which contains 1,260 rounds of human-agent interaction. This dataset is unique because it links a user’s personality scores, their conversational statements, and their actual subsequent actions across multiple rounds and different agent personas.

Key Insights and Ideas

  • Dialogue is a Distractor: Surprisingly, adding dialogue to the decision record provided no detectable gain in overall prediction ranking (AUC). The LLMs were already extracting most of the useful predictive signal from the user's past actions.
  • The "Gullibility" Gap: LLMs are far more optimistic about human honesty than reality warrants. When a user made a promise they hadn't made before, LLMs boosted the predicted probability of cooperation from 13% to 69%. In reality, humans only kept those specific promises 40% of the time.
  • Context Blindness: Humans are social chameleons; we cooperate more with "Agreeable" agents (70.6%) than "Extraverted" ones (54.5%). However, the LLM predictors were almost entirely blind to this, predicting nearly identical cooperation rates regardless of the agent's persona.
  • Averages Conceal Failures: On the surface, adding dialogue seemed to have little effect on accuracy. But looking deeper, the researchers found that dialogue raised accuracy on kept promises while crashing it on broken ones. These two effects canceled each other out in aggregate metrics, hiding a significant failure mode.
LLM predictions vs. actual user behavior. Note how accuracy plummets on broken statements when dialogue is added.

Key Results

  • Baseline Performance: Predictions based on the decision record alone achieved an AUC of 0.80, showing that "deeds" are a strong predictor of future behavior.
  • The Cost of Words: For rounds where users broke their word, adding dialogue caused prediction accuracy to drop from 0.65 to a dismal 0.17.
  • Persona Bias: While users' actual cooperation varied by over 16 percentage points between agent personas, LLM predictions varied by only 1.4 points.
  • Trait Compatibility: Users cooperated more when their own personality matched the agent's (e.g., an agreeable user with an Agreeable agent). The LLMs failed to predict this "compatibility" effect, even when explicitly given the user's personality scores in the prompt.
Figure 1: LLM predictions varied with users’ stated decisions but barely varied with the agent users faced, although users’ behavior did. (a) Each user played a six-round game with each of five agents, each prompted to express a dominant Big Five trait, and could talk with the agent before deciding to cooperate or defect in every round. Eight open-weight LLMs predicted each decision from earlier decisions (deeds) and the round’s dialogue (words). (b) Left: adding dialogue barely changed accuracy on rounds with a stated decision, but raised it on kept statements and lowered it on broken ones. Right: users cooperated more with the Agreeableness than with the Extraversion agent, whereas predictions from all decisions and dialogue barely differed.

Limitations and Future Work

The authors acknowledge several limitations:

  • Population: The study was limited to 42 Chinese university students in a low-stakes environment.
  • Zero-Shot Constraints: The predictors were used in a zero-shot capacity with reasoning disabled. Chain-of-thought or fine-tuning might alter these results.
  • Causality: The study identifies associations but cannot definitively say that the agent's language caused the change in user behavior (as opposed to the agent's decision policy).

Future work should explore "history-aware" or "counterpart-conditioned" predictors that explicitly weigh a user's current promise against their historical "word-keeping" rate. As the authors conclude, for an AI to be a truly effective partner, it must learn to trust, but verify.

Summary updated

Abstract

Web systems increasingly predict users' decisions from what users say, yet when interests partly conflict, users may promise to cooperate to win the other side's cooperation and then not do so. Sales or service agents that rely on such promises may grant concessions to users who then do not keep their promises, at a cost to the party they represent. We test how LLMs combine users' statements with decision records to predict users' decisions, using 1,260 rounds in which 42 users played a repeated prisoner's dilemma-style game with five agents, each prompted to express a dominant Big Five trait, and could talk with the agent before every decision. Across eight open-weight LLMs, predictions from decision records separated cooperation from defection well (AUC = .80), and adding dialogue brought no detectable gain. With dialogue, predictions agreed with users' statements more often, and errors concentrated on rounds in which users broke their statements: accuracy on rounds with a statement barely changed (.73 to .74), while it rose on kept statements (.76 to .95) and fell sharply on broken ones (.65 to .17). Predictions weighted statements more heavily than users' behavior warranted: when users stated a decision they had not yet made in that game, the predicted probability of keeping it rose from .13 to .69, whereas users kept .40 of such statements. Users' decisions also varied with the agent they faced, but predictions hardly reflected this: users cooperated more with the Agreeableness agent than with the Extraversion agent (70.6% vs. 54.5%), whereas predicted cooperation was nearly identical (64.5% vs. 63.1%). Users also cooperated more with an agent when they themselves scored higher on that agent's trait, whereas predicted cooperation barely changed with these scores. Dialogue-based predictions should therefore weigh users' statements by their earlier decisions and the agent they face.
BibTeX
@misc{zhang2026words,
  author = {Baiqiao Zhang and Shuai Ma and Zhifeng Liao and Chao Zhou and Jianshu Wang and Yuhan Zhao and Chaoyue Zhang and Tongjun Si and Yuying Tang and Xiangxian Li and Yong-Jin Liu and Yulong Bian},
  title = {{Words or Deeds? How LLMs Predict Users' Decisions in Human-Agent Dialogue}},
  year = {2026},
  eprint = {2610.36813},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36813},
}
Figure