Abstract: Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, Yejin Choi. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction: The Social Commonsense Gap
While modern AI systems have made significant strides in understanding the physical world—such as knowing that a dropped glass will shatter—they often fail to grasp the nuances of human social interactions. Navigating the "social commonsense" landscape requires Theory of Mind: the ability to infer why someone acted a certain way, how they feel, and what their next move might be.
Existing models struggle with these nuances due to "reporting bias"—social norms are so fundamental to human communication that they are rarely stated explicitly in the text corpora used for training. To address this, researchers introduced SOCIAL IQA, the first large-scale benchmark specifically designed to test an AI’s social and emotional intelligence.
Methodology: Building a Social Benchmark
The authors constructed SOCIAL IQA by leveraging ATOMIC, a knowledge graph of social "if-then" relations. They transformed these structured triples into natural language through a multi-stage pipeline:
- Event Rewriting: Abstract events (e.g., "PersonX spills food") were expanded into realistic scenarios with specific names and contexts.
- Question-Switching Answers (QSA): To prevent models from exploiting stylistic patterns, the researchers used a clever distractor method. They asked crowdworkers to provide correct answers for different questions about the same context. These became the "distractors" for the target question.
- Adversarial Filtering: A stylistic classifier was used to remove examples that were too easy for machines to guess through simple pattern matching.
Key Results and Transfer Learning
The benchmark consists of over 37,000 multiple-choice questions. Initial evaluations revealed a significant performance gap: while humans score between 84-87%, the best-performing model at the time (BERT-Large) reached only 64.5%.
One of the most significant findings is that social knowledge is highly transferable. By "sequentially finetuning" models on SOCIAL IQA before applying them to other reasoning tasks, the researchers achieved new state-of-the-art results on benchmarks like COPA (83.4%), Winograd (72.5%), and DPR (84.0%). This suggests that social intelligence is a foundational component of general reasoning.
The Scale Problem and Future Directions
Error analysis showed that models often struggle with the timeline of events or identifying which participant a question refers to. They frequently rely on lexical associations rather than a true mental model of the social scene.
As shown in the learning curve, current models are far from plateauing, but reaching human-level accuracy would require a dataset nearly 25 times larger than the current 38k examples. Future work aims to move beyond simple data scaling toward models that can explicitly track internal "entity states"—how a person's mood or knowledge evolves over time—and integrate more deeply with structured knowledge graphs.
Summary updated