Giving Language Identity Its Own Attention Channel in Code-Switched Text

People in multilingual communities often switch languages mid-sentence ("main kal office nahi jaunga because I'm sick"). Pretrained models handle this poorly, and the damage concentrates at the switch points: the words where one language hands off to the other. It is also directional. Models stumble more when switching into the lower-resource language than when switching back. This proposal tests a specific architectural explanation and a cheap fix. In a standard Transformer, each attention head compares tokens through one small learned projection, and that projection must encode everything relevant at once: what a word means, where it sits, and which language it belongs to. We propose to move language identity out of that shared projection. Each head gets a separate additive score that depends only on the language of the attending token, the language of the attended token, and whether the two lie in the same monolingual stretch. Because the score is not symmetric, "English attending to Hindi" can be scored differently from "Hindi attending to English."

The idea extends an existing result rather than inventing a new mechanism. A Simple and Effective Positional Encoding for Transformers (DIET) moved position and sentence-segment signals out of the input embeddings and into per-head attention terms. It proved this lifts the head's rank ceiling from $d_h$ to $d_h + d_p$ and stops the two signals from sharing gradient updates. DIET's segments were whole sentences, however, and it never touched code-switched text. Our contribution is to treat the span-level language tag as the segment, give it a directional parameterization, and test directly whether freeing head capacity is what repairs switch-point errors.

Core Claim

We claim that part of the switch-point failure comes from capacity competition inside each attention head. When language identity enters only through token embeddings, a head that needs to match across languages must spend some of its $d_h$ query/key dimensions recognizing language membership. Those dimensions are then unavailable for meaning. A per-head, directional language-pair term absorbs that job at almost no parameter cost, and it should raise accuracy specifically at switch points, especially in the harder direction.

One honesty point shapes the whole design. The textbook low-rank bottleneck of Low-Rank Bottleneck in Multi-head Attention Models shows that a head cannot represent arbitrary attention patterns when its dimension $d_h$ is smaller than the sequence length $n$. That result does not bind on short tweets, where $n$ (often 30 to 60 tokens) is at or below $d_h = 64$. The language-pair term also adds at most rank $K$, the number of languages, which is 2 for a bilingual pair. It therefore cannot cure a true rank deficit. The mechanism we test is subspace competition: the gradient-coupling argument of DIET's second theorem, and the observation in Decoupled Side Information Fusion for Sequential Recommendation that fusing different signals before the projections makes them crowd the same head subspace. The genuine $n > d_h$ regime appears only in our longer-input setting, multi-sentence stories, and we analyze it separately.

Concretely, head $h$ scores token $i$ attending to token $j$ as

$$A^h_{ij} = \frac{q_i^\top k_j}{\sqrt{d_h}} + B^h\big[\ell_i, \ell_j, s_{ij}\big],$$

where $q_i, k_j$ are the usual query and key vectors, $\ell_i, \ell_j$ are the language tags of the two tokens, and $s_{ij}$ indicates whether they lie in the same contiguous monolingual span. $B^h$ is a small learned table that is not constrained to be symmetric. This is exactly DIET's segment term $S Q_s K_s^\top S^\top$ written out for one-hot language segments, extended with a span flag. It adds $2K^2$ numbers per head and can be computed once per sentence.

Research Program

The project starts from a pretrained multilingual encoder, mBERT, which the benchmark papers use. The new tables are initialized to zero, so the model is unchanged at the start. We then continue masked-language pretraining on code-mixed text: the 87,296-tweet Hinglish corpus released with CONFLATOR, and the Marathi-English corpus from My Boli. Finally we fine-tune on downstream tasks.

Language tags come from a separately trained word-level language identifier. This is realistic, since word-level identification reaches 96.6 F1 on English-Hindi in GLUECoS. We report results with predicted tags and also with tags deliberately corrupted at increasing rates, to show how gracefully the method degrades.

The analysis side has two parts. First, for every head we train linear probes that read token language from the query and key vectors, with and without the new term. Second, we compute the entropy-based effective rank of the pre-softmax score rows at switch-point tokens versus non-switch tokens.

The mechanism predicts three things together:

  1. Language becomes less decodable from queries and keys once the table exists.
  2. The dimensions this frees go to meaning, which shows up as better probe accuracy for part-of-speech on the same vectors.
  3. The accuracy gain is concentrated at switch points and in the Embedded-to-Matrix direction that Switch Point biased Self-Training identified as hardest.

If gains appear without the probe shift, the capacity story is wrong even if the method helps, and we will say so.

Baselines

The naive reference point is mBERT given the same continued pretraining and fine-tuning with no architectural change. This isolates what the new term adds beyond simply exposing the model to more code-mixed text. On LinCE, mBERT reaches 63.56 micro-F1 on Spanish-English named-entity recognition, as reported in LinCE.

The strongest existing method on switch-point errors is the switch-point-biased self-training of Switch Point biased Self-Training. It oversamples pseudo-labeled examples near switch points. It reports 66.34 F1 on English-Spanish named-entity recognition and a 5% to 6% reduction in the switch-point part-of-speech gap for English-Hindi and English-Spanish. We rerun it on our backbone with the same unlabeled data and tuning budget. Beating it means exceeding 66.34 F1, and shrinking the switch-point gap by more than it does, both under matched conditions. Because the two methods act on different things, one on data and one on architecture, we also report their combination.

For Hinglish sentiment and translation, the strongest positional competitor is CONFLATOR. It flips the sign of the rotary position rotation at switch points and reports 76.23 F1 on sentiment and 29.1 BLEU on translation. We reimplement its unigram variant on the same backbone and pretraining corpus, so that differences come from the attention design and not the training recipe.

All systems receive the same pretraining tokens and the same 20-trial hyperparameter search per task, and we report means over five seeds. A matched-parameter control spends the table's parameters on a slightly wider head instead.

Three ablations isolate the key ingredient:

  1. Input-level language embeddings. Language-ID vectors are added to the input, like segment embeddings, with everything else identical. This is the decisive comparison for the decoupling claim.
  2. Symmetric table. The table is forced to be symmetric. This tests whether directionality matters, which CONFLATOR's sign inversion cannot express.
  3. Shuffled tags. Language tags are randomly permuted while preserving the number of switches. This tests whether gains come from language information or merely from extra per-head biases.

Why not just…?

Why not just add language embeddings to the input, the way XLM-style models add segment embeddings? This is the real threat. If it matches our method, the decoupling argument has no practical bite. DIET's theorems say input-level addition ties the gradients of language and meaning signals and leaves them competing for the same projection. But those theorems describe a capacity limit, not a guaranteed accuracy loss, and on short sentences with ample capacity the difference could vanish. The input-embedding ablation, together with the probe analysis, settles the question directly. We will report a null result plainly if that is what we find.

Why not just fix switch points with data, through biased self-training or synthetic code-mixed text? Data methods work. Self-training narrows the switch-point gap. However, GLUECoS reports that even with synthetic and real code-mixed training data, natural-language-inference accuracy reaches only 63.1%, and its authors note that augmentation alone may not suffice for semantic tasks. Self-training also over-biases when pseudo-labels pile up. Since the two approaches are complementary, our test is whether the architectural term adds gains on top of self-training, not whether it replaces it.

Why not just use bigger heads, fixing head size as Bhojanapalli et al. recommend? Fixed head sizes help when sequences outrun head dimension. For short tweets, though, the $n > d_h$ condition largely fails, so wider heads mostly add parameters. The matched-parameter wide-head control answers this for tweets. On MULTISO stories, where $n$ does exceed $d_h$, we additionally cross the two interventions, so that a genuine rank effect can be separated from a competition effect.

Why not just use a large language model zero-shot? Multilingual and Code-Switched Sentence Ordering finds GPT-4-class models lose 30% in perfect-match ratio and 11% in Kendall's τ on code-switched stories relative to English. Multilingual Large Language Models Are Not (Yet) Code-Switchers reports systematic underperformance on code-switching. We include one zero-shot large-model reference on the semantic tasks rather than arguing the point.

Proposed Experiments

The main evaluation covers LinCE (part-of-speech, named-entity recognition, and sentiment for Spanish-English, Hindi-English, and Nepali-English) and GLUECoS (natural language inference and question answering for English-Hindi and English-Spanish), plus CONFLATOR's Hinglish sentiment and translation tasks. Every token-level metric is split three ways: switch-point versus interior tokens, by switch direction, and by Code-Mixing Index bucket.

MULTISO provides the long-input regime. Its stories mix sentences from up to five languages, so the language table becomes $5 \times 5$. We measure ordering quality with perfect-match ratio and Kendall's τ. Zero-shot transfer to My Boli Marathi-English tests whether the learned tables carry over to a language pair that was unseen in fine-tuning but present in pretraining.

The decisive check is a joint outcome. The directional decoupled term must beat input-level language embeddings and the symmetric table at switch points, most strongly in the Embedded-to-Matrix direction. It must keep a gain over self-training when the two are combined. And it must coincide with a measurable drop in language information inside the queries and keys. If the shuffled-tag control recovers most of the gain, or the input-embedding ablation matches it, the capacity-competition hypothesis is falsified for this regime.