TL;DR
This paper introduces SocialNLI (SoNLI), the first dataset specifically designed to evaluate how well AI models can infer social nuances like sarcasm and irony from human dialogue. They found that current large language models (LLMs) still struggle significantly with this type of social reasoning, highlighting a critical gap in their "theory of mind" capabilities.
What Problem Does This Paper Address?
Modern AI assistants need to understand human social cues to interact effectively. While LLMs are good at pattern recognition, they often fail to grasp the underlying beliefs, emotions, and intentions that drive human conversations, especially in complex scenarios involving sarcasm or irony. Existing benchmarks don't adequately capture this specific need for inferring social nuances from dialogue, leaving a gap in evaluating and improving models' social intelligence.
Methods
The core of this work is the creation of the SocialNLI dataset. They started with dialogue snippets from the TV show Friends (chosen for its rich social interactions) and used a filtering process to select dialogues that specifically highlight sarcasm and irony. For these dialogues, they generated hypotheses (inferences) and then had human annotators rate the plausibility of these inferences on a scale and provide natural language explanations.
To evaluate models, they employed a counterfactual reasoning approach. For a given inference, models are asked to generate both supporting and opposing arguments. These arguments are then used to calculate a plausibility score. This method aims to assess not just a model's ability to score inferences but also its capacity to generate coherent justifications.
Datasets
- FriendsQA: Used as the source for dialogue transcripts.
- SocialNLI (SoNLI): A newly introduced dataset. It consists of 5,320 dialogue transcripts, each paired with a free-form inference, human-assigned plausibility scores (0-1), and human-written explanations. The dataset is split into an auto/main split (3,920 LLM-generated inferences) and an eval split (1,400 human-scored inferences over 70 dialogues).
Key Insights and Ideas
- LLMs struggle with subtle social cues: Despite advancements, current LLMs still lag behind humans in understanding and reasoning about sarcasm and irony in dialogue. This suggests a fundamental limitation in their "theory of mind" capabilities.
- Dialogue is crucial for social inference: Unlike text-based inference tasks that focus on factual entailment, SocialNLI emphasizes the dynamic and multi-speaker nature of conversations, which is essential for developing robust social AI.
- Counterfactual reasoning is a powerful evaluation tool: Asking models to generate both supporting and opposing arguments for an inference provides a more nuanced way to assess their reasoning abilities than simple classification.
- Human explanations are key: The dataset includes human-written explanations, which are vital for understanding why an inference is plausible or not, and for training models to provide similar justifications.
Key Results
- Model-Human Alignment: In end-to-end counterfactual reasoning, only GPT-40 and GPT-40-mini showed a meaningful positive correlation with human scores. Most other models exhibited weak or negative agreement, indicating they struggle to align their plausibility judgments with human perception.
- Explanation Factuality: When humans fact-checked model-generated explanations, Large Reasoning Models (LRMs) performed slightly better than LLMs, but both groups underperformed human baselines significantly (7.40%-19.01%). This suggests that while models can generate explanations, their factual accuracy and relevance are still lacking.
- Inference Distribution: The SocialNLI dataset is balanced across inference types: concerning reality (37.68%), emotion (31.30%), and belief (31.02%).
Limitations and Future Work
- Dataset Scope: The dataset is derived from a single TV show (Friends) and is English-only, which may limit its representativeness of all social phenomena and perspectives.
- Annotation Cost: The human evaluation process was labor-intensive, leading to a relatively small number of human-annotated data points. The authors suggest this dataset can serve as a diagnostic tool for larger-scale investigations.
- Human Bias: Human annotators inherently have social biases, which could influence the collected data.
- Future Work: The authors aim to use SocialNLI for further research into model weaknesses and to develop solutions for improving social reasoning in LLMs. They also suggest that future work could involve collecting more human annotations to increase redundancy and explore a wider range of social nuances.
Summary updated