Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, Ethan Gotlieb WilcoxarXiv 2024 · OpenReview

TL;DR

The second BabyLM Challenge aimed to improve data efficiency in language models by training them on limited datasets, mimicking human learning. The winning approach combined causal and masked language modeling, showing that architectural and training objective innovations are key, though compute budget still plays a significant role. The multimodal track, however, proved challenging, with no submissions outperforming baselines, highlighting the difficulty of integrating vision and language.

What Problem Does This Paper Address?

Modern language models require massive amounts of data (trillions of words) to train, a stark contrast to how children learn language with remarkable efficiency (less than 100 million words by age 13). This "data-efficiency gap" is the core problem. The BabyLM Challenge seeks to bridge this gap by tasking researchers with training models on significantly smaller, fixed datasets (10M or 100M words) and evaluating them on various linguistic tasks. This encourages the development of more cognitively plausible and data-efficient learning strategies.

Methods

The challenge featured three main tracks:

  • Strict Track: 100M words of text data.
  • Strict-Small Track: 10M words of text data.
  • Multimodal Track: 100M words of text data combined with image data.

Participants could use their own datasets, provided they stayed within the word limits. Evaluation was done using a shared pipeline based on the LM Evaluation Harness, covering tasks like grammatical ability (BLiMP), general language understanding (SuperGLUE), and pragmatic/commonsense reasoning (EWOK). For the multimodal track, additional tasks like Visual Question Answering (VQA) and Winoground were included.

A key methodological innovation this year was allowing participants to bring their own datasets, moving beyond a fixed dataset. This allowed for more exploration into data quality and curation.

Datasets

  • Text-Only Datasets: CHILDES, British National Corpus (dialogue portion), Children's Book Test, Children's Stories Text Corpus, Project Gutenberg, OpenSubtitles, Wikipedia, Simple English Wikipedia, Switchboard Corpus.
  • Text-Image Datasets (for Multimodal Track):
    • 50M words from the BabyLM corpus (stratified sampling).
  • 50M words from image-caption datasets: Localized Narratives (using MS-COCO and Open Images subsets) and Conceptual Captions 3M (CC3M).

The pretraining corpus was updated from the previous year, increasing child-oriented data and transcribed speech, while removing Wikipedia (except Simple English) and QED due to quality concerns.

Key Insights and Ideas

  • Hybrid Architectures Shine: Combining causal language modeling (CLM) and masked language modeling (MLM) objectives, as seen in the winning GPT-BERT submission, proved highly effective. This suggests that leveraging both predictive and reconstructive learning signals is beneficial, especially in low-data regimes.
  • Data is King (Still): The best-performing submissions often involved creating custom training datasets, highlighting that data quality and curation are as crucial as model architecture or training objectives.
  • Compute Budget Matters: Despite the focus on data efficiency, a strong positive correlation between training FLOPs and performance was observed. This suggests that even with limited data, more compute can still lead to better results.
  • Multimodal Learning Remains a Hurdle: The multimodal track saw no submissions outperform the baselines. This indicates that effectively integrating visual information into language models is still a significant research challenge, and models may be prone to unimodal shortcuts.
  • EWOK is Tough: The introduction of the EWOK dataset revealed that current models struggle with commonsense and world knowledge within the 100M word budget, performing near chance. This points to a gap in current pretraining strategies for acquiring deeper understanding.

Key Results

  • Winning Approach: GPT-BERT, a hybrid CLM-MLM architecture, won both the Strict and Strict-Small tracks. It achieved scores of 86.1 on BLiMP and 81.5 on SuperGLUE in the Strict track.
  • Performance vs. Compute: A linear relationship was found between log training FLOPs and average performance across text tasks (β = 2.7, p < 0.01).
  • Multimodal Failure: No submission in the Multimodal track outperformed the baselines (e.g., GIT-1vd125 achieved a macro average of 54.9 on multimodal tasks, while the baseline was 54.8).
  • Human Performance: Top models approached human performance on some tasks (e.g., within 2.5 percentage points on BLiMP), but significant gaps remain, especially on vision tasks and commonsense reasoning (EWOK).
  • Curriculum Learning's Mixed Results: While popular, curriculum learning did not consistently lead to high scores on average, suggesting its effectiveness might be limited in this setting or requires more nuanced application.

Limitations and Future Work

  • Multimodal Challenge: The lack of competitive multimodal submissions highlights the need for further research into integrating vision and language effectively, especially for grounded language learning.
  • World Knowledge Acquisition: The poor performance on EWOK suggests that current pretraining strategies and corpora may not adequately capture commonsense and world knowledge within limited data budgets. Future work could focus on data curation or algorithmic approaches to improve this.
  • Compute vs. Data Efficiency: The strong correlation between FLOPs and performance raises questions about whether current methods are truly data-efficient or simply benefit from more compute. Future challenges might explore ways to decouple these factors.
  • Broader Scope: The authors envision future BabyLM challenges expanding to include other modalities (like speech) and more languages, reflecting the diversity of human language acquisition.

Summary updated

Abstract

The BabyLM Challenge is a community effort to close the data-efficiency gap between human and computational language learners. Participants compete to optimize language model training on a fixed language data budget of 100 million words or less. This year, we released improved text corpora, as well as a vision-and-language corpus to facilitate research into cognitively plausible vision language models. Submissions were compared on evaluation tasks targeting grammatical ability, (visual) question answering, pragmatic abilities, and grounding, among other abilities. Participants could submit to a 10M-word text-only track, a 100M-word text-only track, and/or a 100M-word and image multimodal track. From 31 submissions employing diverse methods, a hybrid causal-masked language model architecture outperformed other approaches. No submissions outperformed the baselines in the multimodal track. In follow-up analyses, we found a strong relationship between training FLOPs and average performance across tasks, and that the best-performing submissions proposed changes to the training data, training objective, and model architecture. This year's BabyLM Challenge shows that there is still significant room for innovation in this setting, in particular for image-text modeling, but community-driven research can yield actionable insights about effective strategies for small-scale language modeling.
BibTeX
@misc{hu2024findings,
  author = {Michael Y. Hu and Aaron Mueller and Candace Ross and Adina Williams and Tal Linzen and Chengxu Zhuang and Ryan Cotterell and Leshem Choshen and Alex Warstadt and Ethan Gotlieb Wilcox},
  title = {{Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora}},
  year = {2024},
  eprint = {2412.05149},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.05149},
}
Figure