Researchers from the Allen Institute for AI (AI2) and the University of Washington have introduced RAINBOW, a new multitask benchmark for commonsense reasoning, and UNICORN, a model that achieves state-of-the-art results across eight different benchmarks. Central to their work is the Cost Equivalent Curve, a novel metric that measures "data efficiency" by calculating exactly how much training data a transfer learning approach saves compared to a single-task baseline.
The Challenge of Universal Commonsense
While large language models (LLMs) have made massive strides, they still frequently stumble on basic "commonsense" tasks—the kind of intuitive reasoning humans perform effortlessly. Research in this area has historically been fragmented, with dozens of individual benchmarks (covering physical reasoning, social interaction, etc.) but no unified framework to build a "universal" commonsense model.
Furthermore, there has been a lack of clear understanding regarding task relationships. Does learning about social etiquette help a model understand physical gravity? How much training data is actually saved by pre-training on one commonsense task before moving to another? The authors address these questions by standardizing benchmarks and creating a more rigorous way to measure transfer learning success.
Methods and Metrics
The paper introduces three major components:
- RAINBOW (REasoning INternally About Objects and Ways): A suite of six diverse commonsense datasets reformatted into a unified multiple-choice format.
- UNICORN (UNIversal COmmonsense Reasoning model): A model based on the T5 (Text-to-Text Transfer Transformer) architecture, specifically utilizing T5-11B to push performance limits.
- Cost Equivalent Curves: Traditional learning curves plot accuracy vs. data size. Cost equivalent curves instead plot the "baseline examples" needed (x-axis) vs. "new method examples" (y-axis) to reach the same accuracy. This allows researchers to see at a glance if a method is more data-efficient: curves below the $y=x$ diagonal indicate success.
The authors compared three transfer "recipes": multitask training (simultaneous), sequential training (source tasks then target task), and multitask fine-tuning (all tasks then target task).
Key Insights and Results
UNICORN established new State-of-the-Art (SOTA) benchmarks across eight popular benchmarks, including $\alpha$NLI (87.3%), CosmosQA (91.8%), HellaSwag (93.9%), and PIQA (90.1%).
Transfer Learning Dynamics
- Sequential Training is Superior: Contrary to the belief that simultaneous multitasking is always best, the authors found that sequential training almost always matches or beats other methods.
- The "Rich Get Richer": Larger models benefit more from transfer learning than smaller ones. While smaller models might struggle to incorporate diverse knowledge without forgetting, larger models (like T5-Large) show significantly better data efficiency gains when using RAINBOW.
- Low-Data Gains: Multitask learning helps most when the target dataset is small. As more target-specific data is added, the relative benefit of having trained on other tasks diminishes.
Limitations and Future Work
The authors acknowledge that T5's original pre-training on GLUE and SuperGLUE may overlap with some tested knowledge, making it difficult to isolate the exact source of "new" commonsense reasoning. Additionally, the failure of generative knowledge graphs (like ATOMIC) to significantly boost QA performance suggests a need for better ways to serialize structured knowledge for transformer models. Finally, the high computational cost of the 11-billion parameter UNICORN limits its accessibility, pointing toward future work in distilled, more efficient universal models.
Summary updated