Social Commonsense Reasoning with Multi-Head Knowledge Attention

Debjit Paul, Anette Frank

Abstract

This paper introduces the Multi-Head Knowledge Attention (MHKA) model, designed to bridge the gap in social commonsense reasoning for large language models. By dynamically generating and attending to semi-structured "if-then" social rules (e.g., mental states and social consequences), the model outperforms strong baselines like RoBERTa on abductive reasoning and a newly introduced task: Counterfactual Invariance Prediction.

Introduction: The Challenge of Social Commonsense

While Pre-trained Language Models (PLMs) are excellent at capturing linguistic patterns, they often struggle with "social commonsense"—the ability to infer the unstated intentions, emotions, and social norms behind human behavior.

The authors identify two main bottlenecks in current approaches: Knowledge Acquisition (obtaining relevant social knowledge not explicitly in the text) and Knowledge Integration (effectively injecting this knowledge into a neural architecture so the model reasons with it rather than treating it as noise).

Figure 1: An illustration of abductive reasoning with social commonsense.

Furthermore, the paper explores the link between abductive reasoning (finding the most plausible explanation for an event) and counterfactual reasoning (predicting if an outcome would stay the same if a previous event were different).

Methodology: Multi-Head Knowledge Attention (MHKA)

The authors propose the MHKA model, which consists of three main modules:

  • Context Encoding Layer: Uses a standard PLM (like RoBERTa) to create representations of the input text.
  • Knowledge Encoding Layer: Instead of using a static knowledge graph, the authors use COMET (a GPT-2 based model trained on the ATOMIC dataset) to dynamically generate contextually relevant social rules covering dimensions like intent, need, and reaction.
  • Reasoning Cell: This core innovation uses a multi-head attention mechanism to allow the model to jointly attend to the input text and multiple pieces of generated knowledge. This allows the model to "emulate" reasoning steps by recursively refining the representation based on different social rules.
Figure 6: The MHKA architecture, featuring the Reasoning Cell.

To prepare the data, the system uses Semantic Role Labeling (SRL) to identify "who did what to whom," ensuring the generated knowledge is correctly mapped to the characters in the story.

Datasets and Evaluation

The model was evaluated on three primary benchmarks:

  1. $\alpha$NLI (Abductive Natural Language Inference): A dataset where the model must choose the most plausible explanation ($H_1$ or $H_2$) to bridge two observations ($O_1$ and $O_2$).
  2. CIP (Counterfactual Invariance Prediction): A newly introduced task where, given a story, if an intermediate event is replaced by a counterfactual one, the model must predict if the final outcome remains "invariant" (unchanged).
  3. ATOMIC: Used via COMET to generate the underlying social commonsense rules.

Key Insights and Results

  • Counterfactuals Aid Abduction: A significant finding is that training to understand counterfactuals improves abductive reasoning. Knowing what wouldn't happen if circumstances changed helps the model identify the most likely cause of what did happen.
  • State-of-the-Art Performance: On the $\alpha$NLI task, RoBERTa-L + MHKA achieved 87.12% accuracy, a significant improvement over the RoBERTa-L baseline (84.48%).
  • Low-Resource Robustness: In scenarios with very little training data (e.g., 1% or 2%), the MHKA model showed even more substantial gains, proving that explicit knowledge helps when data is scarce.
  • Human Validation: Human evaluators confirmed that the model's attention was often focused on the most relevant knowledge rules (approx. 68% of the time), suggesting the model is actually utilizing the provided commonsense.

Conclusion and Future Work

While MHKA effectively integrates social knowledge, it is limited by the quality of the knowledge COMET generates. Future directions include investigating more complex compositional effects—how multiple social rules interact and conflict in more intricate narratives—and reducing the computational overhead of the additional encoding layers.

Summary updated

Abstract

Social Commonsense Reasoning requires understanding of text, knowledge about social events and their pragmatic implications, as well as commonsense reasoning skills. In this work we propose a novel multi-head knowledge attention model that encodes semi-structured commonsense inference rules and learns to incorporate them in a transformer-based reasoning cell. We assess the model's performance on two tasks that require different reasoning skills: Abductive Natural Language Inference and Counterfactual Invariance Prediction as a new task. We show that our proposed model improves performance over strong state-of-the-art models (i.e., RoBERTa) across both reasoning tasks. Notably we are, to the best of our knowledge, the first to demonstrate that a model that learns to perform counterfactual reasoning helps predicting the best explanation in an abductive reasoning task. We validate the robustness of the model's reasoning capabilities by perturbing the knowledge and provide qualitative analysis on the model's knowledge incorporation capabilities.
BibTeX
@inproceedings{paul2020social,
  author = {Debjit Paul and Anette Frank},
  title = {{Social Commonsense Reasoning with Multi-Head Knowledge Attention}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.267},
}
Figure