The role of reinforcement learning in pragmatic reasoning tasks: Modeling and validating the sources of individual differences

John Duff, Alexandra Mayn, Vera DembergPsyArXiv 2025 · DOI

TL;DR

This paper proposes that individual differences in pragmatic reasoning, particularly in "reference game" tasks, stem not from a lack of social competence, but from how people adaptively deploy their reasoning strategies based on experience and resource constraints. Using the ACT-R cognitive architecture, they model this adaptive behavior and validate its predictions with a human experiment, showing that persistence and negative feedback sensitivity are key drivers of performance.

What Problem Does This Paper Address?

The paper tackles a puzzle in the study of pragmatic reasoning, specifically within "reference game" tasks. These tasks are often used to test theories of how we infer meaning beyond literal words, assuming a rational, social reasoning process (Gricean pragmatics). However, empirical results show participants often perform poorly, take many rounds to "get it," and exhibit large individual differences. This raises a question: does this variability challenge the idea of a fundamental, domain-general social competence for pragmatic inference? The authors argue that these observed patterns don't necessarily undermine core competence, but rather reflect how individuals resource-rationally adapt their strategy use in unfamiliar tasks.

Methods

The core of the paper is a computational model built using the ACT-R cognitive architecture. This model simulates how agents might adapt their strategy selection over time. Key elements include:

  • Three interpretive strategies: Literal (A), first-order pragmatic (B), and second-order pragmatic (C), mirroring levels of reasoning in Rational Speech Act (RSA) models.
  • Reinforcement Learning (RL): ACT-R's built-in RL mechanisms are used to update the "utility" of each strategy based on success and failure. This drives adaptation.
  • Individualized RL parameters: The model incorporates parameters for FNEG (strength of negative feedback, influencing disengagement from failing strategies) and persistence (how long an agent continues to explore before guessing). These parameters are varied to simulate individual differences.
  • Simulation and Validation: The ACT-R model is first used to simulate behavior and reproduce patterns from existing literature. Then, a pre-registered experiment is conducted to test the model's predictions by measuring participants' FNEG and persistence on separate tasks and correlating these with their reference game performance.

Datasets

  • Simulated Data: Generated by the ACT-R model, exploring various combinations of FNEG and persistence parameters.
  • Mayn and Demberg (2026) Data: Used for initial model fitting and validation of simulation results.
  • Human Experiment Data: Collected from 149 participants performing a reference game task and two individual difference tasks (Probabilistic Stimulus Selection for FNEG, Anagram Persistence for persistence).

Key Insights and Ideas

  • Resource Rationality as the Key: The paper strongly suggests that individual differences in pragmatic reasoning tasks are primarily driven by how people manage their cognitive resources (effort, time) to adapt to a task, rather than fundamental differences in their underlying social reasoning abilities.
  • Persistence is Crucial: Higher persistence in exploring strategies, even when facing failure, is strongly linked to better performance in reference games. This suggests that "sticking with it" is a significant factor.
  • FNEG's Role is Nuanced: While FNEG (negative feedback sensitivity) is important, its effect might be less direct or more intertwined with persistence than initially hypothesized. The model suggests FNEG's impact is amplified by higher persistence.
  • ACT-R as a Powerful Tool: The paper demonstrates ACT-R's utility not just for modeling cognitive processes, but for simulating and explaining individual differences in performance by varying core RL parameters.
  • Reconciling Theory and Data: The work offers a compelling framework to reconcile the "idealized" Gricean models with messy, variable human behavior, suggesting that performance factors are critical for understanding pragmatic competence.

Key Results

  • Model Simulation: The ACT-R model successfully reproduced known patterns of adaptation and individual differences in reference game performance, particularly the correlation between performance and parameters like FNEG and persistence.
  • Human Experiment:
    • Participants showed significant adaptation over trials, improving accuracy more in simple than complex conditions.
  • Persistence was a significant predictor of reference game accuracy, with higher persistence leading to better performance.
  • A trend for FNEG was observed, but was less robust than persistence.
  • The study found that individual differences in persistence and FNEG were largely independent.
  • Raven's Matrices scores correlated with reference game performance, and this relationship was partially explained by FNEG and persistence.
  • Theory of Mind measures (SST) showed a weaker or non-significant relationship with reference game performance in this study, potentially due to experimental design choices (e.g., feedback).

Limitations and Future Work

  • Model Simplifications: The ACT-R model's strategies are deterministic, and it doesn't fully capture the nuances of probabilistic reasoning or the potential for conditional strategy execution (e.g., using strategy B only when C is too costly).
  • Response Time Mismatches: The model's predictions for response times, particularly the counter-intuitive finding that complex trials might be faster due to guessing, didn't fully align with human data. The authors suggest this could be due to more sophisticated, unmodeled response strategies or automatization.
  • Parameter Sensitivity: The model's predictions for the magnitude of individual difference effects might be overestimated.
  • Future Work: The authors suggest exploring more complex, conditional strategies, investigating the role of other cognitive factors (working memory, processing speed), and extending the model to account for more realistic conversational scenarios and other types of pragmatic inferences (e.g., ad hoc implicatures from overinformative utterances). They also highlight the need for further research into the interplay of task novelty, motivation, and individual differences in adaptation.

Summary updated

Abstract

In Gricean pragmatics, inference during communication is regarded as a form of rational, domain-general reasoning about the intentions of other agents.Studies using the pictorial communication "reference game" task are sometimes used in support of this hypothesis.Yet, measures of pragmatic reasoning in this task sometimes reveal poor performance, with participants requiring many rounds of play before they exhibit theoretically-expected patterns, and demonstrating substantial individual differences in behavior.Do these results challenge the idea of widespread inferencing via fundamental social competence?We advance an alternative proposal here, which posits that these patterns emerge as a factor of the way participants perform pragmatic reasoning in a task: namely, they prefer to use simpler interpretation strategies until experience motivates the use of additional resources.Building off of work modeling task adaptation as reinforcement learning, we use the cognitive architecture ACT-R to simulate the expected behavior of individuals with this kind of resource-rational performance algorithm, subject to individualized parameters for reinforcement learning.These simulations provide a proof-of-concept for our adaptation proposal, recreating known patterns and generating new concrete predictions for the particular domain-general sources of individual variance in reference game tasks.We then go on to validate some of these new predictions in a pre-registered experiment, and find that pragmatic response behavior is indeed related to a participant's general persistence in self-directed exploration of strategies for task completion.Our results offer a path to reconcile difficult empirical data within idealistic models of pragmatic competence.From a broader perspective, we see this as a first step towards a theory of performance factors in pragmatic reasoning, and ultimately, a case study in the value of process-level computational modeling.
BibTeX
@misc{duff2025role,
  author = {John Duff and Alexandra Mayn and Vera Demberg},
  title = {{The role of reinforcement learning in pragmatic reasoning tasks: Modeling and validating the sources of individual differences}},
  year = {2025},
  doi = {10.31234/osf.io/9shv3\_v1},
  howpublished = {PsyArXiv},
}

Loading related content...

Figure