TL;DR
This paper introduces TactfulToM, a novel benchmark designed to evaluate Large Language Models' (LLMs) ability to understand and reason about white lies in real-world conversations. The study reveals that even state-of-the-art LLMs significantly underperform humans in this nuanced aspect of Theory of Mind, highlighting a critical gap in their social reasoning capabilities.
What Problem Does This Paper Address?
Theory of Mind (ToM) is the cognitive ability to infer mental states in ourselves and others, crucial for effective social interaction. While LLMs have shown progress in structured ToM tasks, understanding more complex social nuances like "white lies" – intentional falsehoods told to spare feelings or maintain social harmony – remains a challenge. Existing benchmarks for white lies are limited in scale and conversational depth, and there's a risk of data contamination from established psychological tests. This gap is particularly concerning as LLMs are increasingly deployed in sensitive domains like education and healthcare, where emotional intelligence is paramount. The authors aim to address this by creating a more robust evaluation for LLMs' white lie comprehension.
Methods
The authors introduce TactfulToM, a new benchmark built on a multi-stage human-in-the-loop pipeline. The core of their approach involves:
- White Lie Triplet Decomposition: Breaking down white lies into three components: the "Real Reason" (motivation), the "Lie" (false statement), and the "Truth."
- Role-based Information Asymmetry: Defining four roles (Liar, Accomplice, Observer, Target) with varying access to information about the white lie triplet. This asymmetry is crucial for simulating realistic social scenarios.
- Conversation Generation: Using a human-in-the-loop process with GPT-40 to expand manually crafted seed stories into multi-party conversations. This method aims to avoid biases from direct LLM generation of white lies.
- Hierarchical Evaluation Framework: A three-tier question hierarchy to assess different levels of ToM:
- Info-State Questions: Basic mental state tracking (beliefs, information access).
- White Lie Understanding (1st-Order): Comprehension (identifying the lie) and Justification (understanding the prosocial motivation).
- White Lie Reasoning (2nd-Order): Lie Ability (identifying who can tell a lie) and Lie Detectability (identifying who can recognize deception).
- Comprehensive Evaluation Format: Questions are presented in multiple-choice, binary, list, and free-form formats to ensure robust evaluation and prevent models from relying on superficial cues.
Datasets
The paper introduces TactfulToM, a novel benchmark dataset. It comprises 100 multi-party conversations covering 5 classes of white lies, 3 difficulty levels (based on truth falsifiability), and diverse relationships. The dataset contains 6.7K questions across various formats. This dataset is designed to be more comprehensive and conversational than previous benchmarks like ToMBench.
Key Insights and Ideas
- White lies are a complex frontier for LLMs: Understanding white lies requires more than just tracking beliefs; it necessitates grasping subtle social motivations and emotional nuances.
- LLMs struggle with genuine motivation: While LLMs can identify false statements, they often fail to understand why a white lie was told, particularly the prosocial intent.
- Surface-level detection is insufficient: LLMs may perform well on tasks that can be solved by pattern matching or statistical regularities (e.g., common sense falsehoods), but they falter when deeper reasoning about intentions is required.
- CoT and reasoning models show inconsistent benefits: Current prompting techniques and specialized reasoning models do not consistently improve ToM performance on white lie tasks, suggesting a need for more targeted approaches.
- Information asymmetry is key: The authors' design emphasizes role-based information asymmetry, which is fundamental to white lie scenarios and a crucial aspect for evaluating LLMs' ToM.
Key Results
- Significant human-LLM gap: Even the best-performing LLMs (GPT-40 and DeepSeek families) significantly underperform human accuracy (over 85%) on TactfulToM.
- White Lie Understanding is particularly weak: Models achieve less than 50% accuracy on tasks combining falsity identification with prosocial motivation understanding.
- Info-State vs. White Lie Reasoning: LLMs can track mental states (Info-State questions) but struggle to apply this understanding in white lie contexts (White Lie Reasoning questions), indicating a disconnect between belief tracking and behavioral prediction.
- DeepSeek models show an edge: DeepSeek models generally outperformed GPT-40, especially on justification and Lie Ability questions.
- Common sense scenarios are easier: Models perform better on white lie classes that rely on widely recognized falsehoods or simple symbolic explanations, suggesting a reliance on common sense knowledge rather than deep reasoning.
Limitations and Future Work
- Limited to White Lies: The benchmark focuses exclusively on white lies, not exploring other second-order ToM abilities in depth. Future work could extend this methodology to other non-literal communication forms.
- Lack of Prior Impression: The dataset deliberately excludes prior knowledge between characters, which is common in real-life interactions. Incorporating this aspect would add complexity and realism.
- English-only: The benchmark is currently limited to English. Future work could explore cross-lingual variations to account for cultural differences in communication.
- Ethical Considerations: The authors acknowledge the ethical complexities of deception and the potential for anthropomorphizing AI. They emphasize that the research aims to evaluate LLMs' capabilities, not to advocate for AI that tells white lies.
- Future Directions: The authors suggest that the TactfulToM methodology can be adapted for future research on related ToM tasks. They also highlight the need for specialized methods to handle the complex information structure and multi-role beliefs inherent in white-lie scenarios.
Summary updated