UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark

Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi

Researchers at the Allen Institute for AI (AI2) and the University of Washington have addressed the fragmented landscape of commonsense reasoning in AI by introducing a new multitask benchmark, RAINBOW, and a universal model, UNICORN. Their work provides a roadmap for building AI that understands the "obvious" rules of the world rather than just memorizing patterns.

The Challenge of Commonsense Reasoning

While Large Language Models (LLMs) demonstrate impressive performance, they frequently fail at basic commonsense tasks—everyday knowledge that humans take for granted. The authors identify three primary gaps in current research:

  1. Lack of Generality: Models are often trained on narrow, single-task datasets, leading to overfitting.
  2. Evaluation Blind Spots: Standard accuracy metrics do not quantify how much a model's learning on one task assists its performance on another.
  3. Transfer Uncertainty: The optimal training "recipes" (e.g., multitasking vs. sequential training) for commonsense knowledge were previously unclear.

Methodology: The Cost Equivalent Curve

The researchers utilized the T5 (Text-To-Text Transfer Transformer) architecture, framing every commonsense task as a text-to-text problem. To evaluate these models beyond simple accuracy, they introduced the Cost Equivalent Curve (CEC).

Cost Equivalent Curve for CommonsenseQA: This metric maps model performance to the number of training examples required, showing how many thousands of examples a transfer-learning method "saves" compared to a baseline.

Using isotonic regression, the CEC allows researchers to visualize data efficiency. For instance, on the CommonsenseQA task, the RAINBOW transfer recipe allowed the model to achieve the same accuracy as a baseline while using only half the training data.

The RAINBOW Benchmark and Transfer Strategies

The paper introduces RAINBOW, a suite of six diverse multiple-choice commonsense benchmarks: αNLI (abductive reasoning), CosmosQA (reading comprehension), HellaSwag (event prediction), PIQA (physical commonsense), SocialIQA (social intelligence), and Winogrande (coreference resolution).

The authors compared three primary transfer learning strategies across these tasks:

  • Multitask Training: Simultaneous training on all datasets.
  • Sequential Training: Training on a source collection first, then fine-tuning on the target.
  • Multitask Fine-Tuning: Training on all datasets together, followed by a final fine-tuning step on the target.
Cost Equivalent Curves across RAINBOW tasks: The plots demonstrate that transfer learning strategies, particularly sequential training, consistently improve data efficiency across different benchmarks.

Key Insights and Results

  • Sequential Training is Highly Effective: Training on a broad set of tasks before fine-tuning on a specific target almost always matches or exceeds the performance of training on everything at once.
  • The "Rich Get Richer" Effect: Larger models benefit more from transfer learning. T5-Large showed significantly higher data efficiency gains from RAINBOW than T5-Small, suggesting a synergy between model scale and transferability.
  • The Knowledge Graph Gap: Transferring knowledge from structured graphs (like ATOMIC) to natural language QA is difficult due to "reporting bias"—the fact that common sense is so obvious it is rarely explicitly documented in structured formats.

State-of-the-Art Performance

The resulting model, UNICORN (based on T5-11B), established new state-of-the-art (SOTA) records on eight major benchmarks, including 93.9% on HellaSwag, 91.8% on CosmosQA, and 90.1% on PIQA.

Conclusion

By combining a diverse multitask benchmark with a novel evaluation metric, the authors demonstrate that transfer learning is a powerful tool for improving the data efficiency and general reasoning capabilities of AI models. Future work will likely focus on closing the gap between structured knowledge graphs and natural language reasoning to further enhance these "universal" models.

Summary updated

BibTeX
@inproceedings{lourie2021unicorn,
  author = {Nicholas Lourie and Ronan Le Bras and Chandra Bhagavatula and Yejin Choi},
  title = {{UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i15.17590},
}
Figure