TL;DR
The paper introduces PUMA (Prospective User-state Modeling for Action selection), a novel framework that shifts LLM personalization from passive memory retrieval to active, model-based decision-making. Grounded in the Free Energy Principle (FEP), PUMA infers a user's evolving latent states (like anxiety or readiness to change) and selects dialogue actions that balance uncertainty reduction with goal-directed outcomes.
What Problem Does This Paper Address?
Current personalized dialogue systems primarily rely on memory-augmented or profile-based methods. While excellent at recalling static facts or explicit user history, they fail to capture the hidden, dynamic user states that evolve during a conversation. For instance, if a user says, "I've been taking the new medication for two weeks," the ideal response depends heavily on whether they are anxious about side effects or confidently tracking progress.
Existing systems treat personalization as a retrieval or conditioning problem over observable text. They lack a predictive "world model" of the user, meaning they cannot reason about how a system's response will actively shape the user's future state and behavior.
Methods
PUMA addresses personalization as a sequential decision-making problem under partial observability, leveraging the Free Energy Principle (FEP) and active inference.
The framework consists of three core components:
- User Representation: It separates explicit semantic memory (short-term and long-term) from a latent User State ($s_t$) and a World Model. The world model factorizes into a transition model (predicting how system actions change the user's state) and an observation model (how the latent state manifests as text).
- Free Energy Minimization: At each turn, PUMA updates its variational belief over the hidden user state and refines its world model by minimizing variational free energy.
- Action Selection via Expected Free Energy (EFE): Instead of just generating text, PUMA evaluates candidate dialogue actions by minimizing EFE. This mathematically unifies two competing objectives:
- Epistemic Value (Exploration): Choosing actions to resolve uncertainty about the user's hidden state.
- Pragmatic Value (Exploitation): Choosing actions that steer the user toward preferred, goal-directed states.
Finally, the chosen action guides a response generator to produce the natural-language output.
Datasets
The authors evaluated PUMA on healthcare-oriented counseling and motivational interviewing (MI) benchmarks:
- CAMI (Counseling and Motivational Interviewing): Contains 38 client profiles with complete sessions annotated with per-turn stages of change (precontemplation, contemplation, preparation).
- AnnoMI: A real-world motivational interviewing dataset used to evaluate cross-dataset generalizability of current-state estimation and next-state prediction.
- DynPatient: To evaluate dynamic interactions, the authors built this profile-grounded client simulator using CAMI data to simulate how a patient's readiness state transitions in response to different counselor strategies.
Key Insights and Ideas
- Personalization as Control: Personalization isn't just about remembering the past; it's about actively guiding the future state of the user.
- The Epistemic-Pragmatic Balance: By using EFE, the system naturally knows when to ask clarifying questions to understand the user better (epistemic) versus when to push forward with actionable advice or interventions (pragmatic).
- Decoupling State from Memory: Tracking abstract psychological or behavioral states (e.g., readiness to change) separately from raw text memory allows the LLM to generalize better and avoid repetitive conversational loops.
Key Results
PUMA was tested against strong baseline conversational agents (including CoS and DIIR) using Qwen3-8B and Llama-3.1-8B backbones.
- Dynamic Counseling Effectiveness: As shown in the dynamic evaluation results, PUMA achieved the highest Stage Lift (1.62 for Qwen3, 1.76 for Llama) and Prep Rate (75.9% / 83.0%), meaning it successfully guided simulated patients to the "preparation" stage faster and in fewer turns than baselines.
- Counselor Response Quality: Evaluated via an LLM judge using standard MITI (Motivational Interviewing Treatment Integrity) global scores, PUMA outperformed automated baselines and achieved scores comparable to or exceeding human references in categories like Empathy and Partnership.
- State Tracking Accuracy: In offline tracking, PUMA achieved a current-state accuracy of 0.689 and a next-state prediction accuracy of 0.717, significantly outperforming standard long-prompt baselines.
Limitations and Future Work
The authors note that the dynamic evaluation relies on DynPatient, which is a simulator and inherently an approximation of real human behavior. Consequently, these simulator-based results serve as relative comparisons between algorithmic policies rather than direct evidence of real-world clinical efficacy. Future work includes validating such frameworks in broader settings and exploring transferability across more diverse user-state annotation schemes.
Summary updated