Abstract
Large generative language models have been very successful for English, but other languages lag behind, in part due to data and computational limitations. We propose a method that may overcome these problems by adapting existing pre-trained models to new languages. Specifically, we describe the adaptation of English GPT-2 to Italian and Dutch by retraining lexical embeddings without tuning the Transformer layers. As a result, we obtain lexical embeddings for Italian and Dutch that are aligned with the original English lexical embeddings. Additionally, we scale up complexity by transforming relearned lexical embeddings of GPT-2 small to the GPT-2 medium embedding space. This method minimises the amount of training and prevents losing information during adaptation that was learned by GPT-2. English GPT-2 models with relearned lexical embeddings can generate realistic sentences in Italian and Dutch. Though on average these sentences are still identifiable as artificial by humans, they are assessed on par with sentences generated by a GPT-2 model fully trained from scratch.
Introduction
Training state-of-the-art generative models like GPT-2 is an expensive, data-hungry, and environmentally taxing process. Consequently, high-performing models are often disproportionately available for English. While multilingual models like mBERT exist, they frequently underperform compared to monolingual counterparts. This research investigates whether the structural knowledge embedded within an English model can be leveraged for a new language. The core hypothesis is that the internal "brains" of the model—the Transformer layers—capture universal syntactic or logical patterns that are largely language-agnostic. By updating only the "interface" (the word embeddings), we can effectively recycle the model's intelligence for languages like Italian and Dutch.
Methodology
The authors propose a multi-step "recycling" procedure designed to minimize compute while maximizing performance.
Lexical Embedding Retraining
The English vocabulary is replaced with a new Byte-Pair Encoding (BPE) vocabulary for the target language. Crucially, only this embedding layer is trained while all Transformer layers remain frozen. This constraint forces the new Italian or Dutch embeddings to align themselves with the existing English-trained internal structures.
Scaling via Linear Transformation
To transition from a "small" model (768 dimensions) to a "medium" one (1024 dimensions) without full retraining, the researchers utilize a linear transformation. This maps the learned embeddings from the small model's space to the medium model's space, providing a "warm start" for larger architectures.
Data and Evaluation
The models were trained on substantial corpora: 13.8GB of Italian text (ItWaC) and 13GB of Dutch text (SoNaR). Evaluation metrics included perplexity and human judgments, with comparisons against GePpeTto, an Italian GPT-2 model trained from scratch.
Results and Discussion
The results demonstrate that English Transformer layers are remarkably effective at processing other Indo-European languages.
Performance and Efficiency
The recycled models achieved performance on par with models trained from scratch. For instance, the recycled Italian model's human evaluation scores and perplexity levels were comparable to GePpeTto. Furthermore, efficiency was significantly improved; recycled models reached stable loss values within just 1-2 epochs.
Scaling Success
The least-squares regression method for scaling embeddings proved superior to other techniques like Orthogonal Procrustes or nearest-neighbor approaches. As shown in the table below, the lstsq method achieved the lowest perplexity (PPL) when scaling from small to medium models.
Qualitative Insights
Human judges found the generated text to be highly realistic. While still identifiable as artificial, the recycled models were rated significantly higher than baseline models. The "Power of Alignment" was evident: Italian words naturally gravitated toward the vector space positions of their English equivalents, as seen in the high alignment accuracy for both Italian (85%) and Dutch (89%).
Conclusion and Future Work
This study confirms that "recycling" large language models is a viable and efficient strategy for multilingual adaptation. However, limitations remain. The success of this method may vary for languages with significantly different scripts or syntax (e.g., Chinese or Arabic). Additionally, while the model adapts well to general patterns, specific syntactic nuances still benefit from brief full-model fine-tuning. Future research will explore whether this "recipe" scales to even larger architectures like GPT-3.
Summary updated