Two-Regime Model Merging for Code-Mixed Text

Multilingual language models handle Hindi, Tamil or Spanish well in their standard written form. They do much worse on social-media text that mixes those languages with English inside a sentence and writes the native words in improvised Latin spelling. A common fix is to keep training the model on raw mixed text and then blend its weights back with the original model so it does not forget its base languages. That blend is usually controlled by a single number, and turning it trades code-mixed skill against base-language skill along one curve.

Sensor engineers face the same kind of one-knob trade-off between sensitivity and operating range. Some of them escape it by stacking two layers with different stiffness. A soft layer responds to small inputs, a stiff layer takes over under heavy load, and the ratio of their thicknesses sets where one hands off to the other. This proposal borrows that design and that way of measuring. We blend the adapted model heavily into the network's lower, word-level layers, where spelling noise lives, and blend it lightly or not at all into the upper, meaning-level layers. The depth at which the blend switches is a single tunable handoff point. We then test whether this two-regime blend moves the whole trade-off curve outward, rather than just picking a different point on it.

Core Claim

The claim is narrow. When an adapted code-mixed model is merged with its base model, a merge with a different mixing coefficient for lower and upper layers, split at a handoff depth $d$, should beat every single-coefficient merge on both sides of the trade-off at once: code-mixed task F1 and base-language retention. We also predict where the gain comes from. The two-regime merge should mainly lower the model's sensitivity to spelling variants while keeping its sensitivity to changes in meaning. If the gain instead comes only from shifting the effective average coefficient, the claim is false.

The sensor work supplies the design pattern and the measurement protocol, not any physics. Flexible Pressure Sensor with Tunable Sensitivity and a Wide Sensing Range, Featuring a Bilayer Porous Structure stacks large-pore and small-pore layers made of the same polymer. The large pores collapse first under light loads, which gives high sensitivity (0.063 kPa⁻¹ below 23.6 kPa), and the small pores carry heavy loads out to 654 kPa. The thickness ratio of the two layers sets the transition point. By contrast, simply scaling the thickness of a single layer, as in Wireless Readout System for Pressure Monitoring Using FFF-Printed Mold Fabricated Flexible Piezoresistive Sensors, only slides the device along one trade-off curve. We read a single global merge coefficient as the analogue of single-layer thickness scaling and the two-regime merge as the analogue of the bilayer. The useful part of the analogy is that it forces us to report the whole response curve, not a single operating point.

Why the Layers Should Split This Way

Prior code-mixing work motivates the split. Spelling failures are concentrated at the bottom of the network. Reed at SemEval-2020 Task 9 describes how a transliterated word such as achha is broken by the tokenizer into meaningless pieces. TueMix at SemEval-2020 Task 9 documents how the many spellings of one word blow up the vocabulary. The error analysis in Advancing Sentiment Analysis in Tamil-English Code-Mixed Texts attributes 22% of errors to phonetic or spelling variants and 15% to mixed scripts. These are word-level failures that lower layers should be able to absorb.

The largest error class there, sarcasm and idioms at 35%, needs intact higher-level semantic competence. That is exactly what continued training on noisy text tends to erode. Adapting Multilingual Models to Code-Mixed Tasks via Model Merging supplies our method. It continues pre-training XLM-R or Llama-3.2-1B on unlabeled code-mixed text and merges the result with the base model using Task Arithmetic, which adds the weight difference scaled by one factor $\lambda$, or TIES-Merging, which trims and sign-resolves that difference. It applies one merge policy to all layers. We keep its continued pre-training and merge operators unchanged and vary only how the coefficient depends on depth.

For a model with layers $l = 1,\dots,L$, base weights $\theta^{0}_l$ and continued-pretraining weights $\theta^{c}_l$, the merged weights are

$$\theta_l = \theta^{0}_l + \lambda_l\,(\theta^{c}_l - \theta^{0}_l), \qquad \lambda_l = \lambda_{\text{low}} \text{ if } l \le d, \;\; \lambda_{\text{high}} \text{ otherwise}.$$

Here $\lambda_{\text{low}}$ and $\lambda_{\text{high}}$ are the lower- and upper-block coefficients and $d$ is the handoff depth. The embedding matrix counts as layer 0. The same per-layer coefficients are applied inside TIES after its trimming step.

Research Program

The work has three parts.

The first part is a sensitivity instrument for code-mixed classifiers, modeled on the gauge-factor curves of sensor papers. Orthographic sensitivity is the rate at which predictions flip when content words are replaced by spelling variants. We generate those variants with the phonetic equivalence classes from Normalization of Transliterated Words in Code-Mixed Data Using Seq2Seq Model & Levenshtein Distance, such as a/o and b/v at zero cost, plus doubled-consonant and vowel-dropping rules. We plot this rate against the number of words perturbed, so it is a curve rather than a single number. Semantic sensitivity is the flip rate when a sentiment-bearing word is negated or swapped for an antonym. Range is retention on monolingual parent-language test sets. A good model is flat on the first curve, steep on the second, and keeps the third. We also release this instrument as a diagnostic.

The second part maps the full trade-off. We sweep the global $\lambda$ from 0 to 1 for Task Arithmetic and TIES on XLM-R and Llama-3.2-1B. Each sweep traces the single-coefficient frontier of code-mixed F1 against retention. We then grid over $(\lambda_{\text{low}}, \lambda_{\text{high}}, d)$ and ask whether the two-regime points lie strictly outside that frontier. We also ask whether the best $d$ is stable across language pairs, which would be the analogue of a transferable thickness ratio.

The third part is a concentration variant, modeled on the funnel geometry in Road Narrow‐Inspired Strain Concentration to Wide‐Range‐Tunable Gauge Factor of Ionic Hydrogel Strain Sensor. That sensor channels deformation into a narrow zone to tune its gauge factor from 1.31 to 9.21. Our version restricts continued pre-training itself to layers $\le d$, so that adaptation is concentrated during training rather than redistributed afterward. We report its results whether or not they favor merging.

Baselines

The do-nothing reference is the base model fine-tuned directly on each labeled code-mixed task. The external anchor is the transliteration-plus-curriculum system of MSR India at SemEval-2020 Task 9. It converts Romanized Hindi back to Devanagari and fine-tunes first on synthetic and then on real code-mixed data. It reports F1 72.58 for modified XLM-R on the English-Hindi subtask, against 69.69 for stock XLM-R and 66.21 for stock mBERT. We reimplement it on our splits; the original used the Bing Translator API, so we substitute an open transliterator and report the swap.

The strongest competitor on this specific question is the single-coefficient merge of Adapting Multilingual Models to Code-Mixed Tasks via Model Merging, in both its Task Arithmetic and TIES forms. Its specific results are cut off in the file, so no reported result was found on Lacuna for its numbers, and we will rerun it ourselves. Fairness requires care, because our method has three hyperparameters against its one. We give every method the same continued-pretraining checkpoint, the same fine-tuning recipe, and the same number of development-set evaluations, so a 30-point budget means 30 global-$\lambda$ values versus 30 sampled triples. Beating it means a two-regime point that dominates the best global-$\lambda$ point on both F1 and retention, under equal search budget, across at least three random seeds.

We add two more competitors. The first is normalization preprocessing from Normalization of Transliterated Words in Code-Mixed Data Using Seq2Seq Model & Levenshtein Distance, which reported a Bengali-English sentiment gain from 80.97% to 82.47% accuracy. The second is a fully per-layer coefficient searched with the same budget. Two ablations isolate the key ingredient. Reversing the roles, with a high upper coefficient and a low lower one, tests whether the direction matters. A random layer split with matched average coefficient tests whether the gain is just a different average.

Why not just…?

Why not just tune the single $\lambda$ more carefully? If the trade-off is a single curve, finer tuning only picks a point on it. Our main experiment is the frontier comparison under equal search budget. If the dense global sweep's best point is not dominated by any two-regime point, the claim fails, and we will say so.

Why not just freeze the upper layers during continued pre-training? This is a real threat, and it is essentially our concentration variant. It might match the two-regime merge at lower cost. Merging has two practical advantages: it reuses one checkpoint across many settings of $d$, and it allows nonzero $\lambda_{\text{high}}$ so that some code-mixed syntax can reach the upper layers. That second point matters because IIITT@DravidianLangTech-EACL2021 argues that mixing disrupts syntax as well as spelling. The head-to-head comparison settles it. If freezing ties, we recommend freezing.

Why not just normalize or transliterate the input? Preprocessing attacks the same spelling errors, but the reported gain is modest (+1.5 accuracy), and it depends on a dictionary or a commercial API for each language pair. It also leaves the model brittle to spellings outside the dictionary. Our orthographic-sensitivity curve on perturbations held out from the dictionary measures exactly that brittleness. We also test the combination, since the two approaches may be complementary rather than competing.

Why not just learn all per-layer coefficients? With small labeled code-mixed sets, a large number of free coefficients risks overfitting the development set. We include this as a baseline. If it wins on held-out test data, the bilayer structure is merely a regularizer, and we report it that way.

Proposed Experiments

We evaluate on SemEval-2020 Task 9 Hinglish and Spanglish sentiment (SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets). Because that test set is about 80% positive, we report macro-F1. We also use DravidianLangTech offensive-language detection for Tamil, Malayalam and Kannada, and Bengali-English sentiment. For retention, the file names no monolingual benchmark, so we will use held-out monolingual sentiment sets in each parent language and state their provenance.

The decisive checks are three. The two-regime merge must dominate the global-$\lambda$ frontier on at least three of the four language settings. Its gain must come with a measurably flatter orthographic-sensitivity curve and an unchanged semantic-sensitivity curve. Both the reversed split and the matched-average random split must fail to reproduce the gain. A secondary check asks whether the best handoff depth $d$ chosen on Hinglish transfers to Tamil-English without retuning, which would show that the handoff point reflects where spelling is processed in the network rather than a fit to one dataset.