HabitusMem: Belief Memory over Evolving User Preferences for LLM Agents

Yifan Lu, Gaoyang Guo, Wen Chen, Ziyun Zou, Jian Wu, Han Zhu, Imran RazzakarXiv 2026 · arXiv

TL;DR

HabitusMem introduces a "belief-state" approach to LLM memory, moving beyond simple retrieval of past interactions to maintaining a structured, evolving model of user preferences. By representing memory as a set of revisable beliefs—each with explicit evidence, scope, and status—it enables agents to navigate contradictory or outdated history and act on what the user wants now.

What Problem Does This Paper Address?

Most current LLM agents treat memory as a storage problem: they use Retrieval-Augmented Generation (RAG) to pull relevant snippets from past chats or maintain summaries of "what happened." However, these systems often fail when user preferences evolve.

If a user says "I hate spicy food" in January but later asks for "extra chili" in a specific Thai restaurant review in February, a standard RAG system might retrieve both, leaving the agent confused. Current systems lack a way to distinguish between a standing preference, a one-off exception, and a preference that has been superseded or retracted. As shown in Figure 1, content-based memory passes on fragments of history, but leaves the difficult task of inference—deciding which preference currently holds—to the agent.

Comparison of content-based memory vs. belief memory. Content-based methods often fail to resolve contradictions or distinguish one-off actions from standing preferences.

Methods

The authors propose HabitusMem, a trained "writer" (implemented as a LoRA adapter for Qwen3.5-9B) that sits between the raw history and a frozen agent. Instead of raw text, it generates structured belief records in the format:
claim | evidence | scope | status

  • Claim: The inferred preference or fact (e.g., "Prefers mid-priced reds").
  • Evidence: Citations of specific past messages that support the claim.
  • Scope: The context where the preference applies (e.g., "only for work trips").
  • Status: Whether the belief is active, superseded by a newer one, or retracted by the user.

The system is trained in two stages. First, a "parent writer" learns the record syntax. Second, the model is trained on four auxiliary tasks:

  1. Belief Inference: Writing beliefs from raw history.
  2. State Update: Revising existing beliefs as new evidence arrives.
  3. Request Projection: Selecting only the beliefs relevant to the current user query.
  4. Action Projection: Selecting beliefs required for specific tool-use constraints.
Figure 2: Overview of HabitusMem. The writer takes the retained raw interactions and the current user query as input and infers belief records, each with its evidence, scope, and status. It then projects only the beliefs that bear on the user query to the agent, which answers or acts on the user query. Here, the bland diet and the late-shift hospital address shape the dinner order, whereas the partner-specific vegetarian preference and the one-off oat-milk instruction are kept but not projected. A single writer learns these steps from four tasks: direct writing, state update, request projection, and action projection.

Datasets

The authors evaluate HabitusMem on two primary benchmarks:

  • PersonaMem-v2: A dataset for personalized question answering involving long interaction histories, changing preferences, and explicit requests to "forget" information.
  • VitaBench 2.0: A benchmark for personalized tool use (e.g., booking travel or ordering food) where preferences are often implicit in past behavior.

Key Insights and Ideas

  • Habitus as Latent State: Borrowing from Pierre Bourdieu’s sociology, the authors model the user as a "habitus"—a set of durable but revisable dispositions. This justifies treating memory not as a database of facts, but as a "belief state" that is updated by every new interaction.
  • Request-Conditioned Memory: One of the biggest performance gains comes from projection. By filtering the memory to only include beliefs relevant to the current request, the system reduces noise and prevents irrelevant past data from "distracting" the agent.
  • Explicit Retraction vs. Deletion: Unlike systems that simply delete old data, HabitusMem keeps retracted preferences but marks their status. This allows the agent to recognize when a user is specifically asking not to do something they used to like, which is a more robust way to handle "unlearning."

Key Results

HabitusMem consistently outperformed established memory systems like RAG, Mem0, and A-MEM across different agent families (Qwen and DeepSeek).

  • Personalized Answering: On PersonaMem-v2, HabitusMem achieved an overall accuracy of 0.537 (with Qwen), roughly 4.8 points higher than the strongest baseline. Notably, it excelled in "Knowledge" scenarios where raw history often interferes with the agent's internal knowledge.
  • Tool Use Consistency: On VitaBench 2.0, HabitusMem achieved the highest "all-success" rate (Pass⁴), meaning it was the most consistent at completing complex tasks across multiple trials. It even outperformed giving the agent the "Full Context" of the history, proving that a curated belief state is more effective than raw data.
Table 1: Personalized QA accuracy on PersonaMem-v2. HabitusMem leads in the majority of scenarios compared to other compact memory systems.

Limitations and Future Work

  • Fixed Input Budget: The system uses a position-based rule to select which parts of the raw history the "writer" sees. Very long histories might still result in lost evidence if it falls outside this window.
  • Imperfect Extraction: While better than baselines, the belief extraction process isn't perfect, especially for highly nuanced or deeply implicit preferences.
  • Task Breadth: The current evaluation focuses on QA and tool use. The authors suggest future work should explore broader applications like personalized email composition and complex scheduling.

Summary updated

Abstract

Personalized LLM agents must act on a user's current preferences, which change across interactions and are rarely stated in full. Existing memory systems store and retrieve content from the interaction history and leave the agent to work out which preferences still hold and where they apply. Such content does not indicate whether a preference is still current or where it applies, so agents often act on outdated or misapplied preferences. We instead treat memory as the assistant's belief about the user. Following Bourdieu's account of habitus as durable but revisable dispositions, we model the user as a latent preference state that every interaction, including the current request, both reveals and updates. The belief summarizes what the history supports about this state, and memory consists of the individual beliefs that bear on the agent's current decision, each with its evidence, scope, and status. We implement this view in HabitusMem, a trained writer that states these beliefs as text records for a frozen agent. Across two agent families, HabitusMem achieves the best results among memory systems on both PersonaMem-v2 and VitaBench 2.0. Controlled analyses show that learned request conditioning drives the answering gain, while auxiliary training tasks improve tool use.
BibTeX
@misc{lu2026habitusmem,
  author = {Yifan Lu and Gaoyang Guo and Wen Chen and Ziyun Zou and Jian Wu and Han Zhu and Imran Razzak},
  title = {{HabitusMem: Belief Memory over Evolving User Preferences for LLM Agents}},
  year = {2026},
  eprint = {2611.25855},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25855},
}
Figure