Team Solomon at SemEval-2020 Task 4: Be Reasonable: Exploiting Large-scale Language Models for Commonsense Reasoning

Vertika Srivastava, Sudeep Kumar Sahoo, Yeon Hyang Kim, Rohit R. R, Mayank Raj, Ajay JaiswalSemEval @ COLING 2020 · DOI

TL;DR

This paper tackles the challenge of commonsense reasoning in AI by developing a system that can identify unreasonable statements and explain why they don't make sense. Leveraging large pre-trained language models like BERT and RoBERTa, their system achieved top rankings in the SemEval-2020 Task 4, demonstrating significant progress in making AI understand the nuances of everyday human knowledge.

What Problem Does This Paper Address?

Natural language understanding (NLU) systems have become increasingly sophisticated, excelling at tasks that rely on explicit linguistic patterns. However, they often struggle with commonsense reasoning – the implicit, everyday knowledge that humans take for granted. This gap means AI can miss obvious absurdities, like a person putting books in a pencil box, because they lack the background understanding of object sizes. This paper addresses the need for AI systems that can not only identify statements that violate commonsense but also articulate the reasoning behind why they are unreasonable, a crucial step towards more robust and human-like AI.

Methods

The core of the Team Solomon system relies on transfer learning from large, pre-trained language models (LMs). They experimented with several powerful LMs:

  • BERT: A foundational bidirectional transformer model, fine-tuned for the task.
  • ALBERT: An optimized version of BERT designed for faster training and reduced memory usage.
  • RoBERTa: A "robustly optimized" BERT, trained on a larger dataset with improved pre-training techniques, which showed particularly strong performance.
  • GPT-2: A generative model, used for the explanation generation task.

The system was developed in a three-stage pipeline, corresponding to the three subtasks of SemEval-2020 Task 4:

  • Subtask A (Validation): A binary classification task where the system determines if a given statement is reasonable or unreasonable. They used fine-tuned BERT-based classifiers, feeding each statement individually and comparing their "unreasonability scores."
  • Subtask B (Explanation - Multi-Choice): Given an unreasonable statement, the system chooses the correct reason from multiple options. This was framed as a multi-way binary classification, where each option was evaluated against the statement using connecting negation phrases (e.g., "This does not make sense because...").
  • Subtask C (Explanation - Generation): The most challenging subtask, where the system must generate a natural language explanation for why a statement is unreasonable. They leveraged GPT-2's generative capabilities, fine-tuning it to produce reasons based on the unreasonable statement and a provided reason.

Datasets

The primary dataset used is the ComVE (Commonsense Validation and Explanation) dataset for SemEval-2020 Task 4. This dataset is structured into three subtasks:

  • Subtask A: Pairs of statements, one reasonable and one unreasonable.
  • Subtask B: Unreasonable statements paired with multiple-choice explanations.
  • Subtask C: Unreasonable statements with referential reasons.

The paper also references other related datasets and tasks in the commonsense reasoning field, such as Winograd Schema Challenge, ROCStories, and SWAG, to contextualize their work.

Key Insights and Ideas

  • Pre-trained LMs are powerful commonsense engines: The success of BERT, ALBERT, and especially RoBERTa, demonstrates that large-scale pre-training on diverse text corpora imbues these models with a significant amount of implicit world knowledge.
  • Task-specific fine-tuning is crucial: While pre-trained models offer a strong foundation, fine-tuning them on the specific ComVE dataset and task structure significantly boosted performance.
  • Input manipulation matters: The way input is presented to the model (e.g., feeding statements individually vs. as pairs, using connecting phrases) had a notable impact on performance.
  • Generation is harder than classification: Subtask C, requiring free-form explanation generation, proved to be the most challenging, with BLEU scores indicating room for improvement compared to classification accuracy.
  • RoBERTa's advantage: RoBERTa consistently outperformed BERT and ALBERT, likely due to its more extensive and robust pre-training.

Key Results

The Team Solomon system achieved strong results across all subtasks:

  • Subtask A (Validation): 96% accuracy with RoBERTa-CS, ranking 4th.
  • Subtask B (Explanation - Multi-Choice): 94% accuracy with RoBERTa-Ens (an ensemble of RoBERTa models), ranking 4th.
  • Subtask C (Explanation - Generation): A BLEU score of 19.34 with GPT-2+BMS, securing 2nd rank.

Limitations and Future Work

The authors acknowledge several limitations:

  • Simple sentence structures in the dataset: The ComVE dataset's relatively simple sentences might not fully challenge the models' ability to generate complex, nuanced reasons, potentially leading to lower BLEU scores than expected.
  • BLEU score limitations: The BLEU metric primarily measures word order precision and doesn't fully capture sentence structure, meaning, or recall, which could be important for evaluating generated reasons.
  • Model limitations: Language models can sometimes generate reasons that are mere negations of the input statement rather than true explanations.

Future work suggested includes:

  • Joint modeling: Training a single model for subtasks A and C to improve reason generation by providing an additional reasonable statement.
  • Improving coherence: Developing systems that can generate more coherent and less superficial reasons, potentially by incorporating external knowledge more effectively.
  • Real-world applications: Integrating commonsense reasoning into chatbots for more sensible conversations and using it to detect satirical content in online news.

Summary updated

BibTeX
@inproceedings{srivastava2020team,
  author = {Vertika Srivastava and Sudeep Kumar Sahoo and Yeon Hyang Kim and Rohit R. R and Mayank Raj and Ajay Jaiswal},
  title = {{Team Solomon at SemEval-2020 Task 4: Be Reasonable: Exploiting Large-scale Language Models for Commonsense Reasoning}},
  booktitle = {SemEval @ COLING},
  year = {2020},
  doi = {10.18653/v1/2020.semeval-1.74},
}
Figure