Code-mixing means switching languages inside one sentence, as in Hinglish tweets written in Latin script. Models for this kind of text improve when a multilingual model is further pre-trained on raw code-mixed text, a step called continued pre-training. The cost is that the model forgets some of the plain Hindi, English or Tamil it already knew. The current fix, from Adapting Multilingual Models to Code-Mixed Tasks via Model Merging, runs continued pre-training to the end and then averages the resulting weights back toward the original model. We propose to block the forgetting while training instead. We keep a small memory of gradients computed on monolingual "anchor" text, one per base language, and refresh it only every $K$ steps. Whenever a code-mixed update would push against one of these stored directions, we remove the conflicting part of the update.
The bet is that the gradients describing what the model knows about standard Hindi or English drift slowly during code-mixed training, so an out-of-date copy is good enough. The idea of reusing out-of-date ("stale") gradients comes from federated learning, where servers cache the last update of offline clients so those clients are not forgotten. Here we use the same idea for a different purpose: to make forgetting protection cheap. The cheapness matters because, when code-mixed data is scarce, every forward and backward pass spent on monolingual replay competes with the scarce signal we actually want to learn.
Core Claim
During continued pre-training on scarce code-mixed text, the gradient a model gets from a fixed monolingual anchor batch stays strongly aligned with its own value from hundreds of steps earlier. That alignment is what makes a rarely refreshed, per-language gradient memory work. The claim is that projecting code-mixed updates away from conflicts with this memory preserves monolingual ability about as well as post-hoc merging. It should also learn more code-mixing behaviour than data replay at the same compute, because projection changes the update only where the two conflict, while replay always adds anchor signal.
The claim can be falsified in two ways. If the alignment decays quickly, the stale memory fails. If compute-matched replay does just as well, the mechanism is not needed.
Where This Sits
Two strands of prior work supply the method. Gradient-memory federated methods cache past updates so absent participants are not forgotten. GradMA treats client drift as catastrophic forgetting and solves a small quadratic program so new updates do not oppose stored ones. Federated Learning with Fair Averaging detects negative cosine similarity between updates and projects them onto each other's normal planes. Federated Learning with Erroneous Communication Links proves that reusing a device's last successful update converges to the error-free optimum for convex smooth losses. All of these were tested only on image classification with small networks, and all of them treat staleness as a nuisance forced by unreliable networks. We treat staleness as a deliberate compute trade-off and move the mechanism to transformer language-model pre-training. Our "absent clients" are base languages that are not being trained on at that moment.
A second strand motivates the problem. Adapting Multilingual Models to Code-Mixed Tasks via Model Merging shows that continued pre-training erodes the base languages, which the authors call the monolingual substrate. They repair this after training with Task Arithmetic or TIES-Merging, and report 2–5 F1 over standard fine-tuning and 1–2 F1 over continued pre-training followed by fine-tuning. Their repair depends on a scaling coefficient that the domain-pre-training literature calls brittle. Our method has a refresh interval $K$ instead, and it acts during optimization, not afterwards.
GradSim also uses gradient cosine similarity to detect interference between languages. It uses that signal offline, to split languages into separate models, and it never looked at code-mixed text. We use the same signal online inside a single model.
Improving Pretraining Techniques for Code-Switched NLP constrains the design. Its masking objective, SWITCHMLM, masks the tokens at language switch points, and its probing finds switch-point information in intermediate layers. We therefore keep SWITCHMLM as the code-mixed objective and keep the memory per layer block, so a conflict in one block does not suppress useful updates in another.
Research Program
Let $g^{(l)}$ be the code-mixed gradient for layer block $l$ at the current step. Let $m_a^{(l)}$ be the cached gradient of the same block on a fixed anchor batch from base language $a$, last computed at most $K$ steps ago. Before the optimizer sees the gradient, every anchor whose stored direction conflicts with the update is projected out:
$$ g^{(l)} \leftarrow g^{(l)} - \sum_{a:\, g^{(l)}\cdot m_a^{(l)}<0} \frac{g^{(l)}\cdot m_a^{(l)}}{\lVert m_a^{(l)}\rVert^2}\, m_a^{(l)} . $$
With two or three anchors (for example English plus Hindi, Tamil or Malayalam), this sequential projection is close to the GradMA quadratic program. We will also run the exact quadratic program as a check. Storing one gradient per anchor per block costs a few copies of the model parameters. For Llama-3.2-1B these copies can live on CPU and be loaded block by block.
The first deliverable is a measurement that does not depend on the method working. It is the staleness curve: the cosine similarity between $m_a^{(l)}$ and a freshly computed anchor gradient, as a function of steps since the last refresh, broken down by layer and language. We will track it across a standard continued-pre-training run.
The second deliverable is the training method itself. We will run it on XLM-R and Llama-3.2-1B, matching the merging paper, using the small code-mixed corpora of Improving Pretraining Techniques for Code-Switched NLP (34K–185K sentences, four language pairs). Where the 5.7M-token Hinglish tweet corpus of Team_Swift is obtainable, we will use it as a larger-data condition.
Baselines
The naive reference point is the base multilingual model fine-tuned directly on each downstream task. On SentiMix Hinglish, the SemEval-2020 Task 9 overview reports 65.4% F1 for its M-BERT baseline. The best system in that overview reaches 75.0% F1, using XLM-R with adversarial training. The second reference is plain continued pre-training with SWITCHMLM followed by fine-tuning. It shows how much code-mixed skill is available, and how much forgetting it causes, when nothing is done to protect the base languages.
The strongest existing method is continued pre-training plus Task Arithmetic or TIES merging, following Adapting Multilingual Models to Code-Mixed Tasks via Model Merging. Its strongest reported number in the file is cross-lingual transfer trained on English-Hindi and tested on English-Tamil, where merged models reach 0.65–0.68 F1 against 0.61–0.63 for standard baselines. It is the strongest competitor because it targets exactly the same forgetting. We will reimplement it on our corpora and models. Its scaling coefficient and pruning threshold get the same number of tuning trials as our refresh interval $K$, all chosen on development sets.
Data replay is the most dangerous simple competitor. It mixes monolingual anchor sentences into the continued-pre-training batches. It gets exactly the extra forward and backward passes our method spends on refreshes ($N/K$ anchor passes over $N$ steps), and the same anchor text.
Two ablations isolate the key ingredient. The first, fresh projection, recomputes the anchor gradient every step ($K=1$). It is not compute-matched; it serves as the upper bound that tells us what staleness costs. The second projects against random directions with the same norms and the same conflict rate. It rules out the possibility that any gain comes from simply shrinking or regularizing updates.
Beating merging means matching or exceeding its code-mixed F1 at no greater loss on the monolingual probes, across five seeds. Five seeds are needed because annotator agreement on Hinglish is only 55%. Beating replay means higher code-mixed F1 at equal monolingual retention.
Why not just…?
Why not just replay monolingual text during continued pre-training? This is the real threat. Replay is simpler, and it may give the same retention at the same compute. Our reason for expecting a difference is the drowning effect: Durrani et al. and MONOCODER show that scarce specialized distributions get diluted when generic data is added. Replay always adds anchor signal, while projection intervenes only when the two directions conflict. That is an argument, not evidence. The compute-matched replay baseline, swept over mixing ratios to trace a retention-versus-code-mixed-F1 curve, settles it. If replay's curve dominates ours, we will report the staleness measurements as the main finding and drop the method claim.
Why not just merge after training, as already shown to work? Merging is cheap and proven. However, it can only interpolate between two endpoints, so it cannot keep code-mixed changes in some directions while blocking them in others during learning. It also carries coefficient brittleness. The head-to-head comparison against tuned TIES and Task Arithmetic, plus the cross-lingual English-Hindi to English-Tamil transfer test, will show whether on-the-fly protection adds anything. We will also try combining the two methods: if they stack, that is itself useful.
Why not just use synthetic code-mixed data or parameter-efficient fine-tuning? Synthetic data does not address forgetting. AfroCS-xs found that unvalidated synthetic data often hurt compared with about 60 human-validated sentences. Männistö found that domain pre-training predicts downstream accuracy far better than the choice of parameter-efficient method. We therefore keep both outside the main comparison. A LoRA-only arm will be reported for completeness.
Proposed Experiments
Downstream evaluation will use SentiMix Hinglish and Spanglish, the Tamil-English and Malayalam-English sentiment sets, and GLUECoS NER, POS, QA and NLI for English-Hindi and English-Spanish. We include NLI because GLUECoS reports a baseline of only 63.1%, which tests whether the method helps the semantic tasks that augmentation has failed to move. We will also run the English-Hindi to English-Tamil transfer setting from the merging paper. Monolingual retention will be measured with WikiAnn NER and Universal Dependencies POS for English, Hindi and Tamil, which are the probes GradSim used, though the file has no reported GradSim numbers on them. We will also measure masked-language-model loss on held-out monolingual text. All reports use macro F1 with five seeds and an equal compute count in forward-backward passes.
The first decisive check is the staleness curve. If the anchor-gradient cosine falls toward zero within a few dozen steps in the layers where conflicts occur, the method cannot be cheap and we stop. The second check is the retention-versus-code-mixed-F1 frontier against replay and merging at matched compute. The third is that $K=1$ fresh projection should beat the random-direction control. If even fresh projection fails to beat that control, gradient conflict is not the cause of forgetting in this setting, which would be an informative negative result for both the merging and the gradient-memory lines of work.
References
- Adapting Multilingual Models to Code-Mixed Tasks via Model Merging
- GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgetting
- Federated Learning with Fair Averaging
- Federated Learning With Erroneous Communication Links
- GradSim: Gradient-Based Language Grouping for Effective Multilingual Training
- Improving Pretraining Techniques for Code-Switched NLP
- Team_Swift at SemEval-2020 Task 9: Tiny Data Specialists through Domain-Specific Pre-training on Code-Mixed Data
- SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets
- A Deep Fusion Model for Domain Adaptation in Phrase-based MT
- MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks
- AfroCS-xs: Creating a Compact, High-Quality, Human-Validated Code-Switched Dataset for African Languages
- A Comparative Study of PEFT Methods for Python Code Generation
- GLUECoS: An Evaluation Benchmark for Code-Switched NLP