Know You Before You Speak: User-State Modeling for LLM Personalization in Multi-Turn Conversation

Jiani Luo, Xiaoyan Zhao, Yang Zhang, Shuyi Miao, Bingbing Xu, Stefan Konigorski, Tat-Seng ChuaarXiv 2026 · arXiv

TL;DR

The paper introduces PUMA (Prospective User-state Modeling for Action selection), a novel framework that shifts LLM personalization from passive memory retrieval to active, model-based decision-making. Grounded in the Free Energy Principle (FEP), PUMA infers a user's evolving latent states (like anxiety or readiness to change) and selects dialogue actions that balance uncertainty reduction with goal-directed outcomes.

What Problem Does This Paper Address?

Current personalized dialogue systems primarily rely on memory-augmented or profile-based methods. While excellent at recalling static facts or explicit user history, they fail to capture the hidden, dynamic user states that evolve during a conversation. For instance, if a user says, "I've been taking the new medication for two weeks," the ideal response depends heavily on whether they are anxious about side effects or confidently tracking progress.

Existing systems treat personalization as a retrieval or conditioning problem over observable text. They lack a predictive "world model" of the user, meaning they cannot reason about how a system's response will actively shape the user's future state and behavior.

Methods

PUMA addresses personalization as a sequential decision-making problem under partial observability, leveraging the Free Energy Principle (FEP) and active inference.

Figure 1: Overview of PUMA. Given a user utterance, the system updates its belief over the hidden user state and maintains a world model of user-state dynamics. It then uses this world model to evaluate candidate actions with expected free energy, selecting actions that reduce uncertainty and guide goal-directed outcomes. Finally, the system produces a natural-language response, which gives rise to a user-state transition and the next user utterance, thereby starting the next interaction cycle.

The framework consists of three core components:

  1. User Representation: It separates explicit semantic memory (short-term and long-term) from a latent User State ($s_t$) and a World Model. The world model factorizes into a transition model (predicting how system actions change the user's state) and an observation model (how the latent state manifests as text).
  2. Free Energy Minimization: At each turn, PUMA updates its variational belief over the hidden user state and refines its world model by minimizing variational free energy.
  3. Action Selection via Expected Free Energy (EFE): Instead of just generating text, PUMA evaluates candidate dialogue actions by minimizing EFE. This mathematically unifies two competing objectives:
  4. Epistemic Value (Exploration): Choosing actions to resolve uncertainty about the user's hidden state.
  5. Pragmatic Value (Exploitation): Choosing actions that steer the user toward preferred, goal-directed states.

Finally, the chosen action guides a response generator to produce the natural-language output.

Datasets

The authors evaluated PUMA on healthcare-oriented counseling and motivational interviewing (MI) benchmarks:

  • CAMI (Counseling and Motivational Interviewing): Contains 38 client profiles with complete sessions annotated with per-turn stages of change (precontemplation, contemplation, preparation).
  • AnnoMI: A real-world motivational interviewing dataset used to evaluate cross-dataset generalizability of current-state estimation and next-state prediction.
  • DynPatient: To evaluate dynamic interactions, the authors built this profile-grounded client simulator using CAMI data to simulate how a patient's readiness state transitions in response to different counselor strategies.

Key Insights and Ideas

  • Personalization as Control: Personalization isn't just about remembering the past; it's about actively guiding the future state of the user.
  • The Epistemic-Pragmatic Balance: By using EFE, the system naturally knows when to ask clarifying questions to understand the user better (epistemic) versus when to push forward with actionable advice or interventions (pragmatic).
  • Decoupling State from Memory: Tracking abstract psychological or behavioral states (e.g., readiness to change) separately from raw text memory allows the LLM to generalize better and avoid repetitive conversational loops.

Key Results

PUMA was tested against strong baseline conversational agents (including CoS and DIIR) using Qwen3-8B and Llama-3.1-8B backbones.

  • Dynamic Counseling Effectiveness: As shown in the dynamic evaluation results, PUMA achieved the highest Stage Lift (1.62 for Qwen3, 1.76 for Llama) and Prep Rate (75.9% / 83.0%), meaning it successfully guided simulated patients to the "preparation" stage faster and in fewer turns than baselines.
Table 1: Dynamic evaluation on CAMI under Qwen3-8B and Llama-3.1-8B backbones. Higher values are better for Lift, Prep, and TrigCov, while lower values are better for Turns.
  • Counselor Response Quality: Evaluated via an LLM judge using standard MITI (Motivational Interviewing Treatment Integrity) global scores, PUMA outperformed automated baselines and achieved scores comparable to or exceeding human references in categories like Empathy and Partnership.
Table 2: Counselor-side quality evaluation using MITI Global Scores on CAMI. We report four MI dimensions and their average score; higher scores indicate better counseling quality.
  • State Tracking Accuracy: In offline tracking, PUMA achieved a current-state accuracy of 0.689 and a next-state prediction accuracy of 0.717, significantly outperforming standard long-prompt baselines.

Limitations and Future Work

The authors note that the dynamic evaluation relies on DynPatient, which is a simulator and inherently an approximation of real human behavior. Consequently, these simulator-based results serve as relative comparisons between algorithmic policies rather than direct evidence of real-world clinical efficacy. Future work includes validating such frameworks in broader settings and exploring transferability across more diverse user-state annotation schemes.

Summary updated

Abstract

Personalized dialogue requires more than recalling explicit user histories: systems also need to infer hidden user states that evolve through interaction and shape appropriate response strategies. Existing memory- and profile-based methods primarily reuse observable user information, offering limited support for modeling user-state dynamics or selecting actions based on how they shape future user states. We propose PUMA (Prospective User-state Modeling for Action selection), a framework grounded in the Free Energy Principle (FEP) that formulates personalization as decision-making under partial observability, centered on an explicit user state model that captures latent user states and their action-conditioned dynamics. At each turn, PUMA maintains a belief over the user's hidden state, refines the user state model for observation generation and action-conditioned state transition, and selects dialogue actions by minimizing expected free energy, balancing epistemic and pragmatic objectives under a unified criterion. This formulation shifts personalization from passive memory retrieval to model-based decision-making over user evolution. We instantiate PUMA on healthcare-oriented counseling and motivational interviewing benchmarks with latent state annotations for rigorous evaluation. Experiments show that PUMA improves long-horizon dialogue outcomes while maintaining strong response quality, and a cross-dataset study demonstrates more reliable user-state estimation and next-state prediction.
BibTeX
@misc{luo2026know,
  author = {Jiani Luo and Xiaoyan Zhao and Yang Zhang and Shuyi Miao and Bingbing Xu and Stefan Konigorski and Tat-Seng Chua},
  title = {{Know You Before You Speak: User-State Modeling for LLM Personalization in Multi-Turn Conversation}},
  year = {2026},
  eprint = {2605.24647},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.24647},
}
Figure