People who speak several languages often switch between them inside one sentence, especially online, and they usually write the non-English language in Latin letters with no fixed spelling. Hindi achha ("good") also appears as acha and achhaas. Pretrained multilingual models handle this text badly. Their vocabularies were built from formal text in each language's own script, so a romanized word gets broken into meaningless fragments. Their accuracy also drops sharply at the exact token where the language changes. A tokenizer-free model that reads raw bytes avoids the fragmentation problem. The catch is that byte sequences are several times longer than subword sequences, and the cost of standard attention grows with the square of length.
This note proposes a specific test of whether a different kind of encoder suits this setting better. The encoder is a bidirectional selective state space model: a recurrent layer whose cost grows linearly with length, which reads the sequence forward and backward, and which decides at each byte how much of its running memory to keep. The claim is not mainly about speed. It is that this encoder carries the right inductive bias for code-switching when data is scarce. A language switch is a local change of regime, and a model that learns per-token "how much to forget" can represent that change directly. Running both directions should also cancel the documented asymmetry between switch directions. Our goal is to test this claim fairly against byte-level and character-aware Transformers trained on exactly the same small data, and to measure accuracy specifically at switch points.
Core Claim
Under a matched, small pretraining budget of code-mixed and romanized monolingual text, we expect a byte-level bidirectional selective state space encoder to beat a byte-level bidirectional Transformer of equal size on tokens near language switch points in LinCE and GLUECoS. We expect the advantage to grow as pretraining data shrinks. We also expect the model's per-byte forgetting parameter to spike at switch points without being trained to, which would make it a usable unsupervised language-boundary signal. A second, falsifiable claim is that the backward scan removes the directional switch-point asymmetry reported for Transformers. If the matched byte-level Transformer equals the state space model at switch points across all data scales, the inductive-bias claim is false. In that case the only remaining argument is efficiency on long inputs, and we would report that.
Why this is not already done
The two closest works build bidirectional linear-time models but never touch multilingual or code-switched text. Vision Mamba runs forward and backward selective scans over image patches. It matches or beats a Vision Transformer (80.3% vs 79.8% ImageNet top-1 for the small models) with 86.8% less memory at high resolution, but it uses only supervised vision training. DeltaFlow replaces attention in a text diffusion model with bidirectional gated delta networks, in two forms: an "alternating" form that flips direction every layer and a "parallel" form that runs both directions in each layer. It reaches lower generated perplexity than full attention (21.228 vs 24.218) on English OpenWebText. These papers supply our block designs. DeltaFlow also constrains our design in two ways. It trails attention by about 2.78 BLEU on WMT14 German-to-English translation, which warns that linear recurrence can lose on cross-lingual alignment. And its parallel form doubles recurrent state memory, which is why we test the alternating form as well.
On the code-switching side, the strongest systems are Transformers repaired with data rather than with architecture. GLUECoS continues masked-language-model pretraining of multilingual BERT (mBERT) on synthetic and real code-mixed text. Switch Point biased Self-Training pseudo-labels unlabeled sentences rich in the weak switch direction. That paper also finds char-BERT, which puts a character CNN in front of BERT, to be the strongest backbone, and it documents that English-to-Hindi switches are harder than Hindi-to-English. The only recurrent baseline in this literature is the old non-selective BiLSTM in LinCE, which trails mBERT by nearly 10 points. That result is often read as closing the case for recurrence. We read it as a statement about 2015-era LSTMs, not about modern selective state space models. Earlier recurrent code-switching work, Winata et al. and Pratapa et al., is causal language modeling with LSTMs. These papers motivate our focus on switch points. Pratapa et al. report that switch-point perplexity stays about 4x the average even after 31M synthetic sentences, and Winata et al. needed bilingual part-of-speech supervision to help at switches.
Research Program
The encoder reads UTF-8 bytes directly, so there is no tokenizer to fragment romanized words. Each layer is a selective state space block. A state space layer keeps a fixed-size hidden vector $h_t$ that it updates once per input. In the selective variant the update depends on the current input:
$$h_t = \exp(\Delta_t A)\, h_{t-1} + \Delta_t B_t x_t, \qquad y_t = C_t h_t,$$
Here $x_t$ is the current byte's representation, $A$ is a learned diagonal decay matrix, and $B_t$, $C_t$ are input-dependent read/write projections. The step size $\Delta_t > 0$ is computed from $x_t$. A large $\Delta_t$ makes the model forget its past and overwrite the state with the current input; a small $\Delta_t$ preserves the past. We use the diagonal parameterization and HiPPO-derived initialization from Diagonal State Spaces are as Effective as Structured State Spaces. That paper reports that performance is very sensitive to the learning rate on $\Delta$, so we sweep that rate explicitly instead of inheriting a default. We build bidirectionality in the two ways the literature offers. The parallel version runs forward and backward scans in every layer and adds their outputs, as in Vision Mamba and DeltaFlow-P. The alternating version flips direction in alternate layers, as in DeltaFlow-A, and uses half the state memory. Words are pooled from byte spans for word-level tagging, with spans taken from whitespace and supplied identically to every byte-level model.
We pretrain from scratch with byte-span masking on a fixed corpus. The corpus combines the 10M-sentence Marathi-English corpus from My Boli, the 31M synthetic English-Spanish sentences from Pratapa et al., naturally sourced Basque-Spanish text from EuskañolDS, and unlabeled training-side text from the target benchmarks. We train at three nested data sizes, roughly 1/16, 1/4 and all of the corpus, so that each architecture yields a data-scaling curve and not a single point.
The analysis has two parts. First, we use the language-identification labels in LinCE to locate every switch point. We then ask whether $\Delta_t$ in the forward scan, the backward scan, or both rises at those points, measured as the area under a precision-recall curve when $\Delta_t$ is used as a zero-shot switch detector. Second, we split switch-point accuracy by direction (embedded-to-matrix vs matrix-to-embedded). We compare forward-only, backward-only and fused models to test the prediction that a switch that is hard when read forward is easy when read backward, since it looks like the opposite switch in that direction. Finally, as a targeted intervention, we add a light auxiliary loss that asks $\Delta_t$ to predict LID boundaries during fine-tuning. This tests whether making the forgetting gate explicitly language-aware helps beyond what emerges on its own.
Baselines
The naive reference point is a word-level logistic regression with character n-gram features, in the spirit of TueMix and Reed. Those papers found such models competitive with fine-tuned multilingual models on SemEval-2020 code-mixed sentiment. We run this baseline on every task. On sentiment it may be hard to beat. Its role is to show how much of the sequence-labeling and inference performance actually requires context.
The strongest existing method on these benchmarks is a pretrained Transformer adapted to code-mixing. We report the published anchors and also rerun them where we can. On LinCE, mBERT reaches 82.93% average accuracy and 63.56 micro F1 on Spanish-English NER (LinCE). Switch-point biased self-training raises English-Spanish NER to 66.34 F1 (Chopra et al.). On GLUECoS, the code-mixed-pretrained mBERT reaches 96.6 F1 on English-Hindi language identification and 63.1% on English-Hindi NLI (GLUECoS). For char-BERT, the file describes it as the strongest backbone in Chopra et al., but no reported result was found on Lacuna for char-BERT's per-task scores, so we will rerun it with the released recipe. These models saw far more pretraining text than our models will. We treat them as an upper reference, not as the fair comparison. Our models are not expected to beat them in absolute terms at our budget.
The fair and decisive comparison is a byte-level bidirectional Transformer encoder with matched parameters, identical pretraining bytes, identical masking objective, the same number of optimizer steps, and the same hyperparameter search budget. We give it 24 trials per architecture, covering learning rate, warmup and dropout, plus the $\Delta$ learning rate for the state space models. A second matched control is a char-BERT-style subword Transformer with a character CNN front-end, trained under the same budget. Ablations isolate each ingredient. A forward-only selective model isolates bidirectionality. A non-selective diagonal state space model with fixed $\Delta$ isolates input-dependent forgetting. A modern BiLSTM of matched size tests whether any bidirectional recurrence suffices. Parallel vs alternating fusion measures the memory/accuracy trade-off.
We will count the proposal as successful if three things hold. The selective bidirectional model must beat the matched byte-level Transformer on switch-point-local accuracy, averaged over three seeds, with non-overlapping 95% bootstrap intervals, in at least the two smallest data regimes. It must not lose on overall accuracy. And the non-selective ablation must not account for the gain.
Why not just…?
Why not just fine-tune char-BERT with switch-point biased self-training? This is the strongest practical recipe today, and for a practitioner with a large pretrained checkpoint it may simply remain the best choice. Our question is different. It asks which encoder makes better use of scarce code-mixed data. Chopra et al. themselves report a "sweet spot" beyond which pseudo-labels over-bias the model, and Pratapa et al. show that synthetic data leaves switch-point perplexity about 4x the average. Data fixes therefore appear to saturate. Self-training is also orthogonal to architecture, so we apply the same biased self-training procedure on top of both our model and the matched Transformer. If the gap disappears once both receive it, we will report that architecture does not matter beyond data.
Why not just use a byte-level Transformer, since tweets are short enough that quadratic attention is affordable? This is the real threat, and we concede its premise. At tweet length, efficiency is not a reason to prefer linear recurrence. That is why the matched byte-level Transformer is the central baseline and the claim rests on switch-point accuracy under scarcity. The data-scaling curves settle the question. If the two architectures track each other at every data size, the inductive-bias hypothesis fails. Efficiency then matters only for conversation-length inputs, which we measure separately on SEAME transcripts.
Why not just use a BiLSTM, which is already linear and bidirectional? LinCE shows the vanilla BiLSTM trailing mBERT by nearly 10 points, and LSTMs cannot be trained with parallel scans. A matched modern BiLSTM is included as an ablation. If it matches the selective model, the claim reduces to "byte-level bidirectional recurrence" and the input-dependent $\Delta_t$ is not doing the work.
Why not just add explicit language-ID tags as input features? Winata et al. showed that explicit bilingual tags help at switches, but they require annotation that most low-resource pairs lack. Our zero-shot $\Delta_t$ switch-detection measurement tests whether the selective model recovers this signal without labels. The auxiliary-loss variant shows how much is gained when labels are available. We also give the matched Transformer the same LID-tag input, which tells us whether tags close the gap by themselves.
Proposed Experiments
The main experiment fine-tunes every model on LinCE (language identification, part-of-speech, NER and sentiment for Spanish-English, Hindi-English, Nepali-English and MSA-Egyptian Arabic) and on GLUECoS (English-Hindi and English-Spanish, including NLI and QA). We report overall scores alongside accuracy on words within two positions of a switch point, split by switch direction, for all three data regimes. BnSentMix serves as an out-of-pair sentiment test, and EuskañolDS as a held-out pair.
The first decisive check is the data-scaling gap at switch points, compared against the matched byte-level Transformer. The second is the directional analysis. If backward and fused models show no asymmetry where forward-only models do, the bidirectional component earns its place. The third is $\Delta_t$ as an unsupervised switch detector, scored against LinCE language-identification boundaries. A precision-recall area well above that of a randomly initialized model would show the forgetting gate has learned language boundaries.
For the efficiency question, we report masked-byte loss at switch points versus overall loss on SEAME conversations. We concatenate utterances into contexts of 512 to 16k bytes and measure throughput and peak memory against the byte-level Transformer. This tests whether longer conversational context helps resolve volatile spellings and switches, and whether only the linear model can afford it. Because DeltaFlow's translation gap suggests recurrence can struggle with cross-lingual alignment, we treat a loss on GLUECoS NLI as an informative negative result, not as noise.
References
- Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model
- DeltaFlow: Noise-Adaptive Bidirectional Gated Delta Networks for Embedded Language Flows
- GLUECoS: An Evaluation Benchmark for Code-Switched NLP
- Switch Point biased Self-Training: Re-purposing Pretrained Models for Code-Switching
- LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation
- Code-Switching Language Modeling using Syntax-Aware Multi-Task Learning
- Language Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data
- Diagonal State Spaces are as Effective as Structured State Spaces
- AfroCS-xs: Creating a Compact, High-Quality, Human-Validated Code-Switched Dataset for African Languages
- EuskañolDS: A Naturally Sourced Corpus for Basque-Spanish Code-Switching
- TueMix at SemEval-2020 Task 9: Logistic Regression with Linguistic Feature Set
- Reed at SemEval-2020 Task 9: Fine-Tuning and Bag-of-Words Approaches to Code-Mixed Sentiment Analysis