UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark

Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi

Researchers from the Allen Institute for AI (AI2) and the University of Washington have introduced RAINBOW, a new multitask benchmark for commonsense reasoning, and UNICORN, a model that achieves state-of-the-art results across eight different benchmarks. Central to their work is the Cost Equivalent Curve, a novel metric that measures "data efficiency" by calculating exactly how much training data a transfer learning approach saves compared to a single-task baseline.

The Challenge of Universal Commonsense

While large language models (LLMs) have made massive strides, they still frequently stumble on basic "commonsense" tasks—the kind of intuitive reasoning humans perform effortlessly. Research in this area has historically been fragmented, with dozens of individual benchmarks (covering physical reasoning, social interaction, etc.) but no unified framework to build a "universal" commonsense model.

Furthermore, there has been a lack of clear understanding regarding task relationships. Does learning about social etiquette help a model understand physical gravity? How much training data is actually saved by pre-training on one commonsense task before moving to another? The authors address these questions by standardizing benchmarks and creating a more rigorous way to measure transfer learning success.

Methods and Metrics

The paper introduces three major components:

  1. RAINBOW (REasoning INternally About Objects and Ways): A suite of six diverse commonsense datasets reformatted into a unified multiple-choice format.
  2. UNICORN (UNIversal COmmonsense Reasoning model): A model based on the T5 (Text-to-Text Transfer Transformer) architecture, specifically utilizing T5-11B to push performance limits.
  3. Cost Equivalent Curves: Traditional learning curves plot accuracy vs. data size. Cost equivalent curves instead plot the "baseline examples" needed (x-axis) vs. "new method examples" (y-axis) to reach the same accuracy. This allows researchers to see at a glance if a method is more data-efficient: curves below the $y=x$ diagonal indicate success.
Figure 1: The Cost Equivalent Curve metric illustrated on CommonsenseQA. Curves below the diagonal indicate that the new method requires fewer examples than the baseline to reach the same performance level.

The authors compared three transfer "recipes": multitask training (simultaneous), sequential training (source tasks then target task), and multitask fine-tuning (all tasks then target task).

Key Insights and Results

UNICORN established new State-of-the-Art (SOTA) benchmarks across eight popular benchmarks, including $\alpha$NLI (87.3%), CosmosQA (91.8%), HellaSwag (93.9%), and PIQA (90.1%).

Transfer Learning Dynamics

  • Sequential Training is Superior: Contrary to the belief that simultaneous multitasking is always best, the authors found that sequential training almost always matches or beats other methods.
  • The "Rich Get Richer": Larger models benefit more from transfer learning than smaller ones. While smaller models might struggle to incorporate diverse knowledge without forgetting, larger models (like T5-Large) show significantly better data efficiency gains when using RAINBOW.
  • Low-Data Gains: Multitask learning helps most when the target dataset is small. As more target-specific data is added, the relative benefit of having trained on other tasks diminishes.
Figure 3: Cost equivalent curves across the six RAINBOW datasets comparing multitask, fine-tune, and sequential transfer recipes.

Limitations and Future Work

The authors acknowledge that T5's original pre-training on GLUE and SuperGLUE may overlap with some tested knowledge, making it difficult to isolate the exact source of "new" commonsense reasoning. Additionally, the failure of generative knowledge graphs (like ATOMIC) to significantly boost QA performance suggests a need for better ways to serialize structured knowledge for transformer models. Finally, the high computational cost of the 11-billion parameter UNICORN limits its accessibility, pointing toward future work in distilled, more efficient universal models.

Summary updated

Abstract

Commonsense AI has long been seen as a near impossible goal -- until recently. Now, research interest has sharply increased with an influx of new benchmarks and models. We propose two new ways to evaluate commonsense models, emphasizing their generality on new tasks and building on diverse, recently introduced benchmarks. First, we propose a new multitask benchmark, RAINBOW, to promote research on commonsense models that generalize well over multiple tasks and datasets. Second, we propose a novel evaluation, the cost equivalent curve, that sheds new insight on how the choice of source datasets, pretrained language models, and transfer learning methods impacts performance and data efficiency. We perform extensive experiments -- over 200 experiments encompassing 4800 models -- and report multiple valuable and sometimes surprising findings, e.g., that transfer almost always leads to better or equivalent performance if following a particular recipe, that QA-based commonsense datasets transfer well with each other, while commonsense knowledge graphs do not, and that perhaps counter-intuitively, larger models benefit more from transfer than smaller ones. Last but not least, we introduce a new universal commonsense reasoning model, UNICORN, that establishes new state-of-the-art performance across 8 popular commonsense benchmarks, aNLI (87.3%), CosmosQA (91.8%), HellaSWAG (93.9%), PIQA (90.1%), SocialIQa (83.2%), WinoGrande (86.6%), CycIC (94.0%) and CommonsenseQA (79.3%).
BibTeX
@inproceedings{lourie2021unicorn,
  author = {Nicholas Lourie and Ronan Le Bras and Chandra Bhagavatula and Yejin Choi},
  title = {{UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i15.17590},
}
Figure