MuSeToM: Multi-Sensory Evaluation of Belief Revision with Textual Cues Across Six Senses

Chunkit Chan, Kawai Chung, Sze Heng Douglas Kwok, Yuxuan Liu, Yauwai Yim, Tianshi Zheng, Haochen Shi, Qing Zong, Wong Kai Chung, Pak Hin Lau, Calvin Yang, Yifan Gao, Xiaoman Pan, Ginny Wong, Simon See, Yangqiu SongarXiv 2026 · arXiv

TL;DR

MuSeToM is a new human-validated benchmark that tests whether Large Language Models (LLMs) can correctly determine when a character should revise their beliefs based on sensory evidence. The study reveals a significant "over-updating" bias: models frequently mistake perceptually salient but irrelevant information for definitive proof, failing to distinguish between what catches an agent's attention and what actually justifies a change of mind.

What Problem Does This Paper Address?

Theory of Mind (ToM)—the ability to reason about the mental states of others—is a cornerstone of social intelligence. While LLMs have shown promise in classical ToM tasks like the "Sally-Anne" false-belief test, these evaluations are often static. They ask what someone believes, but rarely if they should change that belief based on new, potentially ambiguous information.

In the real world, humans don't update their beliefs every time they see something new. We evaluate epistemic sufficiency: is this evidence strong enough to warrant a change? Current LLMs struggle with this. As shown in Figure 1, a model might see a character notice an "Almond Milk" carton nearby and incorrectly conclude the character now believes their pitcher contains almond milk, even if the carton was never poured into the pitcher. This confusion between perceptual salience (what is noticeable) and epistemic sufficiency (what is proof) is the core challenge MuSeToM addresses.

Figure 1: An example in MuSeToM. The model in- correctly treats this cue as sufficient for belief change, whereas the gold reasoning preserves Noor’s prior be- lief.

Methods

The authors developed a rigorous pipeline to create MuSeToM:

  1. Source Selection: They adapted narrative contexts from existing ToM datasets (BigToM and OpenToM).
  2. Multi-Sensory Cue Generation: Using a suite of LLMs (GPT-4o, DeepSeek-R1, Qwen), they generated cues across six sensory modalities: visual, auditory, olfactory, gustatory, tactile, and proprioception (awareness of body position).
  3. Human Validation: Three human annotators filtered and labeled every instance to ensure logical consistency and to categorize cues as:
  4. Positive: Sufficient evidence for belief revision.
  5. Hard Negative: Salient and relevant, but insufficient for a definitive update.
  6. Easy Negative: Entirely irrelevant to the belief.
  7. Evaluation Tasks: Models were tested on belief-change detection (binary) and updated-belief identification (multiple choice) across both "clean" (single cue) and "noisy" (cue + distractor) settings.
Figure 2: Overview of the MuSeToM dataset construction pipeline.

Datasets

  • MuSeToM (New): Contains 2,415 human-validated instances derived from 220 source stories. It is the first benchmark to focus specifically on epistemic belief revision across six senses.
  • BigToM & OpenToM: Used as the foundational narrative sources for story contexts and initial belief states.

Key Insights and Ideas

  • The Over-Updating Bias: Models exhibit a "trigger-happy" approach to belief revision. They are much better at accepting sufficient evidence (True Positive Rate) than they are at rejecting plausible but insufficient evidence (True Negative Rate on hard negatives).
  • Detection vs. Identification: Models are significantly better at detecting that a belief might change than they are at identifying what the new belief actually is.
  • Salience Masks Failure: Aggregate accuracy scores often hide failures. A model might look competent overall, but a modality-level breakdown (Figure 5) shows that visual cues—while having high mean accuracy—often have the lowest rejection rates for insufficient evidence.
  • Chain-of-Thought (CoT) is a Double-Edged Sword: While CoT generally improves performance, for some models (like DeepSeek-R1), it actually increased the tendency to over-update, leading to a "negative filtering gain."

Key Results

The researchers evaluated 18 state-of-the-art models, including GPT-4o, Llama-3.1-405B, and DeepSeek-V3.

  • Human vs. Model Gap: Humans achieved ~94% accuracy on clean detection tasks. The best-performing models, like MiniMax-M2.5 (86.05%) and GPT-OSS-120B (82.53%), still lag significantly behind, especially in multiple-choice identification where scores drop to the 60-70% range.
  • The Salience Gap: As shown in Figure 3, there is a massive gap (up to 74% for Phi-3-Medium) between a model's ability to detect a correct change and its ability to reject a "hard negative" (salient but insufficient) cue.
  • Noisy Settings: Adding a distractor cue (Task 3 & 4) proved to be a major stress test. While some models like GPT-3.5-Turbo actually saw an accuracy increase (likely due to a shift in response bias), others like DeepSeek-V3.1 saw performance plummet from 82.57% to 55.97%.
Figure 6: Summary of direct-prompting results across all tasks.

Limitations and Future Work

  • Textual Proxy for Senses: The benchmark uses textual descriptions of sensory experiences rather than raw multi-modal inputs (video/audio). Future work could extend this to true multi-modal environments.
  • Categorical vs. Probabilistic: MuSeToM treats belief revision as a binary "change/no-change" event. In reality, humans often update their confidence in a belief rather than replacing it entirely.
  • Static Narratives: The stories are fixed. Testing belief revision in dynamic, interactive multi-agent simulations would be a natural next step to see how these biases play out in real-time social coordination.

Summary updated

Abstract

Theory of Mind (ToM), the ability to infer mental states in others, is pivotal for human social cognition. Existing evaluations of ToM in large language models (LLMs) mainly focus on static belief attribution and false-belief reasoning, leaving a critical aspect of social reasoning underexplored: can LLMs decide whether perceptual evidence is sufficient to justify another person's belief revision? To address this gap, we present MuSeToM, a human-validated, text-based benchmark for evaluating epistemic belief revision in ToM reasoning. MuSeToM contains 2,415 instances adapted from BigToM and OpenToM, with textual cues spanning six sensory categories, and tests whether models can determine when a protagonist should revise a belief and infer the resulting updated belief state. Using MuSeToM, we systematically evaluate state-of-the-art LLMs under direct and chain-of-thought prompting, revealing a pronounced dissonance: while models often detect that a belief may change, they struggle to identify the updated belief and frequently over-update on perceptually salient but epistemically insufficient cues.
BibTeX
@misc{chan2026musetom,
  author = {Chunkit Chan and Kawai Chung and Sze Heng Douglas Kwok and Yuxuan Liu and Yauwai Yim and Tianshi Zheng and Haochen Shi and Qing Zong and Wong Kai Chung and Pak Hin Lau and Calvin Yang and Yifan Gao and Xiaoman Pan and Ginny Wong and Simon See and Yangqiu Song},
  title = {{MuSeToM: Multi-Sensory Evaluation of Belief Revision with Textual Cues Across Six Senses}},
  year = {2026},
  eprint = {2611.16662},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16662},
}
Figure