Social IQa: Commonsense Reasoning about Social Interactions

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, Yejin Choi

Abstract: Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, Yejin Choi. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction: The Social Commonsense Gap

While modern AI systems have made significant strides in understanding the physical world—such as knowing that a dropped glass will shatter—they often fail to grasp the nuances of human social interactions. Navigating the "social commonsense" landscape requires Theory of Mind: the ability to infer why someone acted a certain way, how they feel, and what their next move might be.

Existing models struggle with these nuances due to "reporting bias"—social norms are so fundamental to human communication that they are rarely stated explicitly in the text corpora used for training. To address this, researchers introduced SOCIAL IQA, the first large-scale benchmark specifically designed to test an AI’s social and emotional intelligence.

Methodology: Building a Social Benchmark

The authors constructed SOCIAL IQA by leveraging ATOMIC, a knowledge graph of social "if-then" relations. They transformed these structured triples into natural language through a multi-stage pipeline:

  1. Event Rewriting: Abstract events (e.g., "PersonX spills food") were expanded into realistic scenarios with specific names and contexts.
  2. Question-Switching Answers (QSA): To prevent models from exploiting stylistic patterns, the researchers used a clever distractor method. They asked crowdworkers to provide correct answers for different questions about the same context. These became the "distractors" for the target question.
  3. Adversarial Filtering: A stylistic classifier was used to remove examples that were too easy for machines to guess through simple pattern matching.
Figure 1: An example of the SOCIAL IQA task, illustrating how different questions (next action vs. prior need) are used to test social reasoning.

Key Results and Transfer Learning

The benchmark consists of over 37,000 multiple-choice questions. Initial evaluations revealed a significant performance gap: while humans score between 84-87%, the best-performing model at the time (BERT-Large) reached only 64.5%.

One of the most significant findings is that social knowledge is highly transferable. By "sequentially finetuning" models on SOCIAL IQA before applying them to other reasoning tasks, the researchers achieved new state-of-the-art results on benchmarks like COPA (83.4%), Winograd (72.5%), and DPR (84.0%). This suggests that social intelligence is a foundational component of general reasoning.

The Scale Problem and Future Directions

Error analysis showed that models often struggle with the timeline of events or identifying which participant a question refers to. They frequently rely on lexical associations rather than a true mental model of the social scene.

Figure 5: The learning curve for SOCIAL IQA suggests that nearly 1 million training instances would be required to reach human-level performance.

As shown in the learning curve, current models are far from plateauing, but reaching human-level accuracy would require a dataset nearly 25 times larger than the current 38k examples. Future work aims to move beyond simple data scaling toward models that can explicitly track internal "entity states"—how a person's mood or knowledge evolves over time—and integrate more deeply with structured knowledge graphs.

Summary updated

BibTeX
@inproceedings{sap2019social,
  author = {Maarten Sap and Hannah Rashkin and Derek Chen and Ronan LeBras and Yejin Choi},
  title = {{Social IQa: Commonsense Reasoning about Social Interactions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=dqUL_mnQM0Y},
}
Figure