Non-Contiguous Character Pooling for Zero-Shot Transfer to Templatic Languages

Tokenization-free language models read raw characters and shorten the sequence early, usually by pooling each block of four adjacent characters into one vector. That works for languages that build words by gluing pieces end to end. It fails for Arabic-style morphology, where a word's core meaning sits in a few consonants scattered through the word and the grammar sits in the letters placed between them. We propose to replace the fixed adjacent-block pooling with a pooling step that reads the word and decides which character positions to gather into each compressed vector. The positions can skip over intervening letters, so one vector can hold a discontinuous root. The compressed sequence then goes to a global model whose cost grows linearly with length, and we test whether the change narrows the zero-shot gap: the drop in accuracy when a model fine-tuned only on English is evaluated on Arabic and other morphologically unusual languages.

The case for this is concrete. On MLQA, XLM fine-tuned on English reaches 68.0 F1 on Spanish but 54.8 on Arabic (MLQA). Controlled translation experiments show that neural models can handle non-concatenative morphology almost perfectly when the input representation exposes it; they fail when the segmentation hides it (Amrhein & Sennrich). The best character-level encoders still hard-code contiguity at their compression step. The question is whether moving that one step off the contiguity assumption buys cross-lingual accuracy specifically where morphology is non-concatenative, and nowhere else.

Core Claim

The claim is narrow and falsifiable. In a character-level multilingual encoder, the downsampler (the layer that shortens the character sequence before the deep model) is where the contiguity assumption does its damage. A downsampler whose gathering positions depend on the input should improve English-to-Arabic zero-shot question answering over an otherwise identical contiguous downsampler. The improvement should be concentrated in languages with stem-internal morphology. It should vanish for concatenative languages. It should also vanish for Amharic, a Semitic language whose script fuses each consonant with its vowel into a single character, so the root's discontinuity is invisible at the character level.

That last prediction is what separates this from "more capacity helps." If the gains are uniform across languages, or appear on Amharic, the morphological story is wrong.

One point about the input must be stated up front, because it shapes the whole design. Written Arabic usually omits short vowels. The root ك-ت-ب appears as كتب, كاتب (with a long-vowel letter inserted), مكتوب (a pattern prefix and an inserted و), or اجتمع-style forms with an infixed ت. The characters that interrupt the root are therefore weak letters (the vowel-like letters alif, waw and ya) and pattern consonants. They are not the written short vowels. De Roeck & Al-Fares found this empirically: their hand-built root clustering downweights weak letters to 0.25–0.50 and uses a bigram that skips one intervening character. On unedited newspaper text that design reaches 94.04% clustering accuracy on one set, and 84% on infixed forms where the baseline gets 0%. Our downsampler is a learned, content-dependent generalization of that skip.

Mechanism

Let $h_i$ be the vector for character $i$ after a light contiguous local layer. With downsampling rate $r$, compressed slot $j$ has anchor $s_j = r j$. A small network looks at the contiguous window around $s_j$ and predicts $K$ fractional offsets $o_{j,k}$, bounded to $\pm W$ characters. The slot's vector is

$$z_j = \sum_{k=1}^{K} W_k \, \mathrm{interp}\big(h,\; s_j + o_{j,k}\big),$$

where $W_k$ are learned weight matrices shared across slots, and $\mathrm{interp}(h, p)$ linearly interpolates between $h_{\lfloor p \rfloor}$ and $h_{\lceil p \rceil}$ so that the offsets receive gradients.

The interpolation trick is taken from Dilated Convolution with Learnable Spacings (DCLS). That paper makes kernel tap positions continuous and trainable in 2D vision convolutions, gaining +0.4 to +0.6 ImageNet top-1 on ConvNeXt. The key difference is that DCLS positions are fixed after training, the same for every input. A fixed spacing cannot follow a root across templates, because the gap between root consonants differs between كتب, كاتب and مكتوب. We therefore predict offsets from content, and we run fixed-spacing DCLS as the ablation that isolates this choice.

Three implementation details guard against known problems. Offsets are initialized to the contiguous positions $s_j, s_j+1, \dots$, so training starts from the CANINE-style solution. The interpolation kernel is widened early in training and narrowed over time, because linear interpolation only passes gradients between adjacent characters. $W$ is set to 8 characters, which spans a word plus attached clitics.

The compressed sequence goes into a bidirectional selective state space model. A state space model is a recurrent layer that keeps a fixed-size memory updated in parallel across the sequence. In the selective (Mamba) variant, the update gates depend on the current input, so the model can choose what to retain. Our encoder runs one scan forward and one backward, since an encoder needs context from both sides. This backbone plays a supporting role rather than a central one. MambaByte showed that selective state space models handle raw byte streams in linear time, but it evaluated only on English books, arXiv and code. Whether such a model forms the shared cross-lingual representation that makes zero-shot transfer work is untested. Long Wikipedia passages in TyDi QA are also exactly where quadratic attention over characters becomes expensive. For span extraction we reuse CANINE's upsampling: the compressed outputs are concatenated back onto the character-level states.

Relation to Prior Work

Three existing systems fix the vocabulary problem but keep contiguity.

CANINE hashes characters, runs local attention, applies a 4× strided convolution over adjacent characters, and then a 12-layer Transformer. Its best variant also uses subword boundaries to define the masking objective during pretraining.

Charformer scores soft candidate blocks of 1–4 adjacent characters, and its authors explicitly state that the blocks are contiguous.

ByT5 removes the vocabulary imbalance across scripts but feeds full-length byte sequences to a quadratic Transformer.

We keep CANINE's overall shape and its output path, change only the downsampler and the backbone, and pretrain with purely character-level span masking, so that no subword bias re-enters.

Prior morphology work motivates the proposal and also constrains it.

Amrhein & Sennrich is the main motivation. It also contains the strongest warning: very fine segmentation (BPE-500) recovers much of the loss, reaching 94% on reduplication against 72% for BPE-32k. Character input with enough context may therefore already suffice. That risk shapes our baselines.

JZR and the ISRI stemmer show that root extraction is feasible but noisy. The reported accuracies are 51.88% and 61.63% on a 1,200-word gold set. JZR also loses 11.5% relative accuracy when infix handling is removed, which is direct evidence that skipping over internal material matters.

FLOTA repairs subword fragmentation, but only for concatenative morphology.

TyDi QA supplies the evaluation rationale. Its native questions have low lexical overlap with the answer passages (0.26 for Arabic versus 1.29 for XQuAD), so the model must match different inflections of the same root instead of copying strings.

Research Program

We pretrain four encoders of roughly BERT-base size (within ±2% parameters) on multilingual Wikipedia in 104 languages, following CANINE's corpus. Each sees the same number of characters and uses the same character span-masking objective. The four form a 2×2 design: contiguous strided downsampler versus content-offset downsampler, crossed with a Transformer backbone versus a bidirectional Mamba backbone. Three further models share the budget:

  • a fixed-spacing 1D DCLS downsampler with the Mamba backbone;
  • a contiguous downsampler whose window width equals the offset range $r + 2W$, matching the content-offset model's reach without its selectivity;
  • a flat bidirectional character-level Mamba with no downsampling.

Every model is fine-tuned only on English: English TyDi QA GoldP and SQuAD for question answering, and English CoNLL NER. We then evaluate zero-shot on all TyDi QA languages, on MLQA (both same-language and mixed-language settings), and on MasakhaNER Amharic.

For the typological analysis, languages are grouped a priori:

  • Templatic: Arabic.
  • Stem-mutating or partly reduplicating: Finnish (consonant gradation), Russian, German (umlaut, in MLQA), and Indonesian (reduplication).
  • Largely concatenative or isolating: English, Spanish, Swahili, Vietnamese, Chinese, Japanese, Korean, Thai, Telugu, Bengali, Hindi.
  • Fused-script Semitic control: Amharic.

We also run two representation diagnostics outside the question-answering pipeline. The first is root retrieval. We embed Arabic words from the JZR gold sample and from the De Roeck newspaper sets, then measure how often a word's nearest neighbours share its root. The ISRI and gold labels define the ground truth, and Hebrew serves as an unlabeled qualitative check. The second is an offset audit. For Arabic words with gold roots, we measure what fraction of the gathered interpolation mass lands on root consonants versus weak letters and pattern letters, compared with the chance fraction under contiguous gathering.

Finally, we use Segtest as a small controlled study. We train from scratch on WMT16 German-English with a small decoder, and only the downsampler differs between runs. This checks whether the mechanism helps on reduplication and vowel harmony when there is no cross-lingual confound.

Baselines

Naive reference. mBERT with its fixed WordPiece vocabulary, fine-tuned on English under the same protocol. Published context: minimal-answer F1 of 50.5 overall on TyDi QA (TyDi QA), and failure on Amharic because the script is missing from its vocabulary (CANINE). We use its public checkpoint, not a re-pretrained one, so it measures where off-the-shelf systems stand. It is not a controlled comparison.

Strongest existing method. CANINE-C, reproduced as our contiguous-downsampler, Transformer-backbone cell under the matched pretraining budget. CANINE reports beating mBERT by 2.8 F1 on passage selection (CANINE-S) and by 5.7 F1 on the minimal-answer task (CANINE-C with n-gram features) on TyDi QA. Those numbers come from its own training setup. For English-only zero-shot TyDi QA with CANINE, no reported result was found on Lacuna, so our reproduction is the comparison point. On MLQA the reference is XLM's reported 54.8 F1 on Arabic and 48.8 on Hindi (MLQA).

Fairness protocol. Every system gets the same fine-tuning data, the same grid of three learning rates by two epoch counts selected on English development data only, and three fine-tuning seeds.

What counts as beating CANINE-C. Under the same backbone, the content-offset downsampler must improve Arabic F1 on both TyDi QA and MLQA by more than twice the seed standard deviation. It must also lose no more than 0.5 F1 averaged over the concatenative group.

Ablations. The fixed-spacing DCLS model and the wide contiguous window isolate the key ingredient, which is content-dependent position choice. They separate it from non-contiguity alone and from receptive-field size alone.

Flat character Mamba. This tests whether compression is needed at all. We compare it on accuracy and also on wall-clock throughput, since MambaByte reports that uncompressed byte models run up to 2.2× slower than subword ones.

Language-specific root features. ISRI root identifiers are appended to the character embeddings, for Arabic only. This is a hand-built upper-reference, not a competitor for zero-shot use.

Why not just…?

Why not just use a character model with a wide enough contiguous window? This is the most serious threat. A window of 20 characters contains every root consonant, and Amrhein and Sennrich found that fine segmentation already recovers most of the lost accuracy, with BPE-500 reaching 94% on reduplication (Amrhein & Sennrich). What a contiguous window cannot do is exclude the intervening pattern letters. It must mix them in with fixed per-position weights, even though their positions differ from one template to the next. The width-matched contiguous baseline tests exactly this. If it matches the content-offset model on Arabic and on root retrieval, we will conclude that reach, not selectivity, was the missing ingredient, and we will report that outcome.

Why not just port DCLS to 1D with learned but fixed spacings? Fixed spacings encode one gap pattern per channel. Arabic templates change the gaps between root consonants, so a fixed kernel can at best cover a few templates in separate channels. The fixed-DCLS ablation measures how much of the gain survives without input-dependent offsets. If it matches the full model, the simpler design should be preferred.

Why not just add a root extractor or stemmer as preprocessing? The best available extractors are wrong on roughly 40–50% of words: ISRI at 61.63%, JZR at 51.88% (Fixing the Infix). They are also language-specific, which conflicts with a single model that transfers zero-shot to scripts it was never tuned for. We still run ISRI features on Arabic. If this crude hand-built signal beats the learned downsampler, the learned mechanism is not finding roots, and the offset audit should show that directly.

Why not just run MambaByte-style flat selective recurrence, which can already skip characters? A selective state space model can shrink its update step on a weak letter and effectively carry the previous consonant forward. So a flat model might learn discontinuous roots with no special downsampler. The cost is processing every character through the deep stack. The flat character Mamba baseline settles the accuracy side. The throughput measurement settles whether compression earns its place. If the flat model is both as accurate and fast enough, the downsampler matters only for efficiency, which is a weaker claim.

Why not just keep a Transformer backbone? The morphological claim does not depend on the backbone, which is why the design is factorial. The Mamba cells answer a separate question the literature leaves open: whether a fixed-memory linear model supports zero-shot transfer at all. Reporting the downsampler effect under both backbones prevents that question from confounding the morphological one.

Decisive Checks

The idea stands or falls on four outcomes.

The first is the typological pattern of gains. The content-offset minus contiguous difference, under each backbone, should be largest for Arabic. It should be smaller but positive for Finnish, German, Russian and Indonesian, and indistinguishable from zero for the concatenative group and for Amharic. A gain on Amharic would indicate generic capacity rather than morphology.

The second is the offset audit. On gold Arabic roots, the gathered mass on root consonants should clearly exceed the contiguous-chance fraction. Failing that, the mechanism is doing something other than what we claim, whatever the downstream scores.

The third is the root-retrieval diagnostic. Same-root nearest-neighbour rates should move in the same direction as downstream Arabic accuracy.

The fourth is the Segtest control. The content-offset downsampler should raise accuracy on reduplication and vowel harmony toward the near-100% abstract-representation ceiling reported there, without hurting compounding and circumfixation.

If Arabic improves only under the Transformer backbone, or only on MLQA and not on TyDi QA's low-overlap native questions, we will report a qualified result rather than the full claim.