Introduction
RecToM is a novel benchmark designed to evaluate whether Large Language Models (LLMs) can effectively "read the room" during recommendation dialogues. Moving beyond simplistic logic puzzles, it assesses an AI's Theory of Mind (ToM)—its capacity to infer a user's latent mental states (beliefs, desires, and intentions) and utilize this information to predict or judge strategic conversational moves.
The Challenge of Social Reasoning in CRS
Conversational Recommender Systems (CRS) depend heavily on understanding the nuances of human interaction. To be truly effective, a recommender must employ Theory of Mind: for instance, if a user remarks, "I've seen Elf but I love Home Alone," the system should infer a preference for nostalgic comedy, even without explicit instruction.
Existing ToM benchmarks for LLMs often suffer from two primary limitations:
- Oversimplification: Many rely on physical perception tasks like the "Sally-Anne" test, which lack the social complexity of real-world conversations.
- Retrospective Bias: Most tests focus on explaining past events rather than proactive reasoning—using inferred mental states to determine future actions.
RecToM addresses these gaps by placing LLMs in realistic, asymmetric recommendation scenarios where preferences and intentions evolve dynamically over multiple turns.
Methodology: The BDI Framework
The researchers developed RecToM based on the Belief-Desire-Intention (BDI) model of mental-state attribution. The benchmark evaluates two primary dimensions:
- Cognitive Inference: Identifying the internal states of participants. This includes Belief Reasoning (tracking the recommender's perception of seeker preferences), Intention Reasoning (identifying the goal of a specific utterance), and Desire Reasoning (predicting user interest in a suggestion).
- Behavioral Prediction: Applying inferences to future actions. This involves Prediction Reasoning (guessing the next strategy) and Judgement Reasoning (evaluating strategy effectiveness).
The benchmark utilizes a Question-Answering (QA) format with varied complexity to simulate real-world decision-making cognitive loads.
Dataset and Multi-Granular Taxonomy
RecToM was constructed using the REDIAL dataset, a corpus of movie recommendation dialogues. The authors refined 336 high-quality dialogues, resulting in a benchmark of 20,524 QA pairs across 10 question types.
A key innovation of RecToM is its multi-granular intention taxonomy. Unlike previous datasets, it distinguishes between broad categories and specific sub-intentions (e.g., differentiating a general "Request" from a specific "Ask for more recommendations").
Key Insights
- The Sycophancy Problem: LLMs exhibit a strong "Yes-Man" bias. When asked if a strategy would be effective, models predicted "Yes" approximately 93% of the time, prioritizing user "pleasing" over accurate judgment.
- Granularity and Complexity: While LLMs perform adequately on coarse-grained tasks, their accuracy drops significantly on fine-grained sub-intentions. Performance also degrades as the "choice space" expands from binary to multiple-choice options.
- The Limits of Chain-of-Thought (CoT): Interestingly, CoT prompting provided only marginal gains and occasionally hindered performance. This suggests that social intuition may not align with the linear logic that CoT typically aids in technical domains.
Results and Performance
Testing across state-of-the-art models (including GPT-4o and DeepSeek-v3) revealed a significant gap between AI and human performance (82–98% accuracy).
- Top Performer: DeepSeek-v3 (with CoT) led in Belief Reasoning at 79.46% accuracy.
- The Bottleneck: Judgement Reasoning proved the most difficult, with models often performing below random guess levels due to sycophantic bias.
- Fine-grained Struggle: On fine-grained intention tasks for the recommender role, the model average was a mere 27.74%.
Conclusion
RecToM highlights that while LLMs possess some ToM capacity, it remains fragile and prone to social biases like sycophancy. While currently focused on the movie domain, the benchmark sets a foundation for evaluating social reasoning. Future work will likely move toward dynamic evaluations in live dialogues and developing de-biasing techniques to improve strategic judgment.
Summary updated