Assistants that remember long conversations increasingly need to answer questions about other people, not only about the world. Examples include "Does Maya know the dinner moved to Friday?" or "Why did Sam bring the old keys?" Answering these requires a theory of mind, meaning the ability to model what someone else knows, believes, or wants, as distinct from what is actually true. Current long-term memory systems store a single objective record. When a fact changes, they overwrite or link the new version, and every stored image or utterance is available to answer any question. That design invites a known failure. When a language model holds the full truth, it projects that knowledge onto characters who lack it, a failure called the "curse of knowledge." On short text stories, GPT-4o states a character's knowledge correctly yet predicts that character's behavior at 49.5%, near chance, and judges the behavior at 15.3% (SimpleToM). Images make the leak worse. A picture showing the true state of the world is strong evidence that the model's attention latches onto even when the character never saw it.
We propose to store, alongside every memory, who had access to it and how: present in the session, visible in or looking at the image, or told by someone else. A question about a person's belief is then answered by running the reasoner only on that person's access-scoped slice of memory, so the model cannot use evidence the person never had. The proposal turns three adjacent problems into one testable claim. Hiding the true state, rather than asking the model to ignore it, should remove a specific class of hallucination: confidently attributing to a person knowledge or perceptions that their history does not support. We call this perspective hallucination. Answering "what does B believe now" across sessions is also a multi-hop problem. It requires following chains of telling: C saw it in session 2, told A in session 5, and A mentioned it to B in session 9.
Core Claim
In long, multi-session, multimodal dialogue, belief and behavior questions fail mainly because memory is perspective-blind, not because retrieval misses or the reasoner is too weak. The decisive comparison is between giving the model the access information as text and actually withholding the inaccessible evidence. Withholding should matter most when that evidence is an image showing the true state, because visual evidence is what the model can least resist. If telling the model the access tags works as well as filtering, the claim fails, and the contribution shrinks to an annotation scheme plus a benchmark. We will report that outcome as such.
Relation to Prior Work
Two lines of work motivate the gap. SimpleToM separates explicit knowledge inference from applied behavior prediction on two-sentence synthetic stories. It shows that a ToM-specific prompt reminding the model of the character's mental state recovers accuracy, with Claude-3.5-Sonnet reaching 97.1%. That result constrains us. Our method only earns its place where the reminder cannot be written easily, which is when the model must itself reconstruct, from hundreds of turns and images, who knew what. MindDial tracks first-order, second-order, and common belief turn by turn in short negotiation games. We borrow its recursive belief levels but attach them to a persistent memory store rather than a per-session module, and we extend them to visual percepts.
A second group supplies the memory machinery, and in each case we change the same thing. SimpleMem compresses dialogue into timestamped atomic facts with mixed lexical, dense, and symbolic retrieval. Its finding that normalizing time at ingestion matters (a 56.7% temporal-reasoning drop when removed) is the model for our access normalization at ingestion. AriadneMem builds a temporal graph that merges conflicting facts with update edges and finds multi-hop bridges with Steiner-tree search. Both systems treat memory as objective. Merging "2 PM" into "3 PM" erases the fact that one participant still believes 2 PM. We keep per-agent versions instead.
The ATM-Bench study (According to Me) constrains the design in two ways. Structured schema records beat prose descriptions by about 20% under oracle retrieval, and flat "piled" memory often matched graph memory because noisy visual extraction corrupts graph edges. We therefore use flat schema records with an access field rather than a committed graph. Mem-Gallery shows that captions are lossy but that raw images dumped into context hurt more than text alone. V-Mem routes retrieval by modality but returns whole rounds that distract small models. Both define the representational constraint: images stay as pointers to crops and are loaded only when they fall inside the target agent's access.
On the multimodal side, MMToM-QA shows that GPT-4V reaches 44% while BIP-ALM, which extracts symbolic states and runs Bayesian inverse planning, reaches about 76%. This supports structured perspective modeling but relies on a brittle simulator parser and no dialogue memory. MoMentS finds that video models do worse with full context than with focused clips and fail most on Percepts and Beliefs. Access scoping is a principled way of choosing the "focused clip" per question. Hallucination work supplies the mechanism story. Why LVLMs Are More Prone to Hallucinations in Longer Responses shows attention drifting from image tokens to generated text in long outputs. SHARP steers middle-layer states to suppress text-prior hallucination. Summary-Guided Decoding applies summaries only to image-related tokens. All three treat hallucination as drift away from the image. Our case is the reverse: the model is too faithful to an image the character never saw. Decoding-time fixes aimed at increasing visual fidelity should therefore not help, and could hurt. POPE supplies the evaluation template: balanced yes/no probes with adversarial negatives.
Research Program
Ingestion turns each dialogue round and each image into schema records. Following ATM-Bench, each record holds time, location, entities, content, and an image-crop pointer. Each record also gets an access field: a set of (agent, mode, source) entries. Mode is one of heard, said, saw, or told. Text access comes almost free from session participation and addressee cues. Visual access is predicted by a vision-language model asked who is present in the image and who is facing the relevant object, with a confidence score. Records below a confidence threshold are marked uncertain rather than assigned. Telling propagates access: when a record says that A told B about event e, B receives told-access to A's version of e, not to the ground truth. Updates never overwrite. Each agent's latest accessible version is kept.
At query time a light parser identifies the target agent and the belief order. For a first-order question, retrieval runs over records in the target's access set. For a second-order question ("what does A think B knows"), the system keeps records that A has access to and for which A has evidence of B's access. Retrieval and answering then use an unchanged backbone, with SimpleMem-style retrieval over the scoped subset. Images are loaded only if in scope. Objective questions bypass scoping entirely, so standard memory performance should be unchanged.
The benchmark is the main build cost, because no benchmark in the file combines long multi-session dialogue, images, and information asymmetry. We will construct one by extending Mem-Gallery and LoCoMo conversations. In each, we introduce a third participant with partial session attendance. We inject state changes, some shown only in images, and some relayed through chains of one to four tellings. We then generate belief, behavior-prediction, and behavior-judgment questions in the SimpleToM style. Chain length is controlled so the multi-hop dimension can be measured directly. A human-validated subset will check question quality and provide gold access labels. Alongside accuracy we report a perspective-leak rate: the fraction of answers asserting content outside the target's gold access set. We also run a POPE-style probe ("Did B see the umbrella?") with adversarial negatives, where the object is present in the image but B was absent.
Baselines
The naive reference point is the full-context baseline: the whole multimodal history concatenated into a long-context vision-language model, with a plain answer prompt. It is fair because it has strictly more information than our system. Our system beats it if the scoped version wins on belief questions while matching it on objective ones. On LoCoMo, SimpleMem reports this baseline at 18.70 average F1 against its own 43.24 (SimpleMem). This tells us full context is already weak on objective memory.
The strongest existing method is SimpleMem's pipeline with the ToM-specific prompt from SimpleToM added at answer time, plus V-Mem-style modality routing for images. SimpleMem is the best reported text memory system in the file. V-Mem reports 0.82 LLM-judge on Mem-Gallery against 0.56 for Omni-SimpleMem (V-Mem). The ToM prompt is the intervention that closed the gap on short stories. This combination is the honest competitor: good retrieval plus good perspective prompting. It will use the same backbone, the same retrieval budget in tokens, and the same number of prompt-tuning trials on a shared development split. We beat it if our method is at least 10 points higher on belief and behavior-judgment accuracy on the new benchmark, with a lower leak rate, and the gap grows with telling-chain length and with image-only state changes. No reported result exists for any method on this new benchmark, so these thresholds are ours.
The key ablation is tags-in-prompt: identical memory and access tags, but the tags are printed next to each record instead of used to filter. This isolates withholding from merely informing. A second ablation compares oracle access tags with predicted tags, which bounds how much the visual access predictor costs. A third removes telling-propagation, keeping only direct access, to measure the multi-hop component. For the existing multimodal ToM benchmarks, the reference numbers are BIP-ALM's ~76% and GPT-4V's 44% on MMToM-QA, and LLaVA-Video-72B's 67.6% on MoMentS.
Why not just…?
Why not just use the ToM prompt that already reached 97.1% on SimpleToM? This is the real threat. On two-sentence stories, the prompt works because the relevant mental state fits in one line that the prompt can restate. In a 300-turn history with images, the model must first work out who attended which session and who was in which frame. That reconstruction is exactly where the curse of knowledge operates. The strongest baseline includes this prompt, and the chain-length analysis will show whether its advantage decays as access becomes harder to reconstruct. If it does not decay, the problem is easier than we think, and we will say so.
Why not just record session participants and tell the model? That is the tags-in-prompt ablation, and it may recover much of the gain on text-only asymmetry. Our prediction is that it falls short specifically on image-only state changes. There, the image showing the truth sits in context and pulls the answer, the inverse of the attention drift described in Why LVLMs Are More Prone to Hallucinations in Longer Responses. If tags-in-prompt matches filtering even there, withholding is unnecessary, and the claim reduces to "annotate access."
Why not just apply a hallucination fix such as SHARP or Summary-Guided Decoding? These methods push the model toward the image. Perspective hallucination comes from using an image the character could not see, so the fix points the wrong way. We will run SHARP on the backbone as a check. We predict it lowers object hallucination on POPE but raises the leak rate on our adversarial "present but absent observer" probes.
Why not just use a long-context model on everything? MoMentS, Mem-Gallery, and the LoCoMo full-context number above all show that more context hurts in these settings. The full-context baseline is included to confirm this on belief questions specifically.
Decisive Checks
Three results settle the proposal. The first is the gap between filtering and tags-in-prompt on image-only state changes; we expect filtering to win clearly here. The second is the slope of accuracy against telling-chain length for our method versus the strongest baseline. The third is that objective questions on standard LoCoMo and Mem-Gallery stay within noise of SimpleMem and V-Mem, so that scoping costs nothing when it should not apply. On MMToM-QA and MoMentS, we test whether access-scoped clip selection narrows the Percepts and Beliefs deficit without a simulator-specific parser. We report the predicted-tag versus oracle-tag gap as the main limitation. If visual access prediction is too noisy, the ATM-Bench lesson about brittle extraction applies to us as well, and the method's value will depend on how cheaply access can be logged at capture time.
References
- SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMs
- Theory-of-Mind Enhanced Dialogue Generation in Situated Contexts
- SimpleMem: Efficient Lifelong Memory for LLM Agents
- AriadneMem: Threading the Maze of Lifelong Memory for LLM Agents
- According to Me: Long-Term Personalized Referential Memory QA
- Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents
- V-Mem: Modality-Routed Retrieval for Long-Term Multimodal Agentic Memory
- MMToM-QA: Multimodal Theory of Mind Question Answering
- MoMentS: A Comprehensive Multimodal Benchmark for Theory of Mind
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- SHARP: Steering Hallucination in LVLMs via Representation Engineering
- Mitigating Hallucinations in LVLMs via Summary-Guided Decoding
- Evaluating Object Hallucination in Large Vision-Language Models