TL;DR
This paper introduces BAMBINO-LM, a novel continual pre-training strategy for small language models that mimics human bilingual language acquisition. By using a "parent" model to provide feedback via Proximal Policy Optimization (PPO) and alternating this with standard causal language modeling, BAMBINO-LM significantly improves a model's ability in a second language (Italian) while maintaining its proficiency in the primary language (English).
What Problem Does This Paper Address?
The BabyLM challenge aims to train small language models on datasets that mimic the linguistic exposure of a child. This paper extends this by investigating how to equip these models with bilingual capabilities, drawing inspiration from how immigrant children re-acquire their heritage language through interaction with parents. Specifically, they address the question: can a language model initially trained on a majority language (English) be effectively continually pre-trained on a minority language (Italian) by leveraging feedback from a second, fluent model acting as a "parent"? This is crucial for developing more versatile and human-like language models that can navigate multilingual environments.
Methods
BAMBINO-LM employs a two-phase continual pre-training strategy:
- Causal Language Modeling (CLM): This is the standard next-token prediction task used to train decoder-only models. The model learns to predict the next word in a sequence based on the preceding words.
- Proximal Policy Optimization (PPO) with Parent Model Feedback: This is the core innovation. The "baby" model generates text, and a "parent" model (a larger, pre-trained Italian language model) provides feedback. This feedback is in the form of a perplexity-based reward, indicating how well the baby model's output conforms to the parent model's understanding of Italian. PPO is used to update the baby model's parameters based on this reward, encouraging it to generate more "parent-like" Italian.
A key architectural innovation is the alternating run strategy, where the CLM and PPO phases are interleaved. This mimics the dynamic interaction between a child and parent, where learning and feedback occur in a back-and-forth manner. This alternation is crucial to prevent undesirable behaviors like repetitive word generation.
Datasets
The paper uses an Italian dataset created to be comparable in size to the "strict-small" track of the BabyLM challenge. This dataset was preprocessed identically to the BabyLM corpus. For evaluation, they use:
- UINAUIL: Four Italian language tasks.
- GLUE: Four English language tasks.
- BLiMP: A benchmark of linguistic minimal pairs for English, used in the original BabyLM challenge.
Key Insights and Ideas
- Human-inspired bilingual acquisition: The core idea of mimicking parent-child interaction for heritage language acquisition is a powerful analogy for language model training.
- PPO as a "parental" feedback mechanism: Using a larger, fluent model's perplexity to guide a smaller model's learning in a second language is an effective way to transfer knowledge and improve performance.
- Alternation is key: Simply combining CLM and PPO isn't enough. The alternating nature of the training phases is critical for preventing the model from simply mimicking and repeating the parent model, leading to more robust learning.
- Cross-lingual transfer trade-offs: The observed degradation in English performance after Italian pre-training mirrors human second language acquisition, where focus on L2 can sometimes temporarily impact L1 fluency. This suggests that language models, like humans, may face similar learning trade-offs.
Key Results
- Significant Italian improvement: BAMBINO-LM achieved an average improvement of 0.1197 on Italian downstream tasks (UINAUIL) compared to the BabyLM baseline.
- Minimal English impact: English classification tasks (GLUE) showed no significant change in performance.
- Expected English degradation: A slight decrease was observed on the English BLiMP dataset (0.6255 to 0.5503), which the authors attribute to cross-lingual transfer effects similar to human learners.
- Ablation study confirms effectiveness: Removing either the PPO feedback mechanism or the alternating strategy significantly decreased Italian performance, highlighting the importance of both components.
Limitations and Future Work
- Reliance on a base model: The approach requires an existing model pre-trained in the majority language (English in this case).
- Language validation: The paper notes that while Italian datasets were used, explicit validation of the language of the text was not performed.
- Hyperparameter tuning: The PPO model's parameters were not experimentally tuned.
- Perplexity as a metric: Using perplexity as a reward metric might be influenced by many factors of the parent model, suggesting potential for improvement.
Future work includes exploring alternative metrics and reward learning mechanisms that better align with human feedback, potentially capturing more nuanced linguistic quality and providing constructive corrections. They also plan to validate the method on languages more distant from English and investigate the potential for alternating with the first language during training.
Summary updated