Efficient language models increasingly split their work between two parts. One is a cheap local operator: exact attention restricted to a small window of neighboring tokens. The other is a global operator that summarizes the whole sequence, such as linear attention or a few full-attention layers. Every such design sets the local window as a fixed token count, usually 64 to 128, chosen to fit GPU hardware. A token, though, means very different things in different languages. Under a shared multilingual tokenizer, a Sinhala or Slovak word may be split into many fragments. Under a tokenizer trained on that language alone, the same word may be one or two tokens. A 64-token window can therefore cover a whole clause in one setting and part of a single inflected word in another.
This note proposes that, in bidirectional encoders (models that read the whole sentence at once, like BERT), the quality of a local-plus-global design depends on how many words the local window covers, not how many tokens. If that holds, it has a practical consequence. A good language-specific tokenizer should allow a smaller window and fewer expensive global layers at equal accuracy, so better tokenization and efficient architectures compound. If it fails, and quality depends only on token count, then hardware-driven window sizes transfer unchanged across languages. That would also be worth knowing.
Core Claim
Let $f$ be tokenizer fertility, the average number of tokens per whitespace word in a language, and let $k$ be the local window size in tokens. A bidirectional sliding window of $k$ tokens covers about $w = k/(2f)$ words on each side of the current position.
The claim has three parts. First, downstream accuracy of a hybrid bidirectional encoder, plotted against $w$, should fall on one curve across tokenizers and languages. Plotted against $k$, it should not. Second, the number of full-attention layers needed to match a full-attention encoder should shrink as $f$ falls. Third, those full-attention layers are most valuable in the upper layers.
The first part is a curve-collapse prediction. The factor of two appears because a bidirectional window is split between left and right context. Nothing in the literature predicts the shape of the curve. We only predict that a single curve exists.
Grounding and Position Relative to Prior Work
The efficient-attention side supplies the method. BASED pairs a Taylor-approximated linear attention with 64–128-token exact softmax windows. It shows that this small window closes most of the recall gap to full attention, and it outperformed Mamba by 10.36 points on recall-intensive tasks at 1.3B parameters. LoLCATs uses a 64-token window when converting pretrained Transformers to linear attention. It attributes the window's value to recovering the "spiky" attention patterns that linear attention smooths away, and it reports that later layers are harder to linearize. Transformer-LS combines a segment-wise window with a low-rank global projection inside each layer. Its dual normalization is needed to balance the two streams. All three are evaluated only on English, so a fixed token window has only ever been tested at roughly English fertility. That is the untested assumption this proposal targets.
The bidirectional side supplies the global operator. LION gives an exact bidirectional linear attention with a decay mask and a chunkwise form. It matched BERT-Large on C4 masked language modeling while training 3× faster. Hydra builds bidirectional state-space mixers and reached 84.3 average GLUE against BERT-Base's 83.5. Hydra also notes that such structured mixers are weaker than softmax attention at exact retrieval. DeltaFlow is the closest existing hybrid bidirectional model. It interleaves bidirectional gated delta layers with occasional full attention for text diffusion. It uses only English and English–German data with fixed tokenization, and it still trails full attention by about 2.78 BLEU on WMT14.
We reuse LION's bidirectional linear attention as the global path. We borrow DeltaFlow's interleaving idea, but we study encoders rather than diffusion. Our new variable is tokenizer fertility, which DeltaFlow never varies.
Theory constrains the design. A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attention proves that no number of linear layers can replace missing full-attention layers on multi-step composition. We therefore never test hybrids with zero full-attention layers as a competitor for reasoning-like tasks. We treat the number of full-attention layers as a quantity to measure.
The tokenization side motivates the question. SlovakBERT reports that 33% of its vocabulary is used on Slovak text, against 4% of XLM-R's. Its layer probes show part-of-speech information peaking in middle layers and sentence similarity in the final layers. This is the empirical basis for placing full attention high in the stack. Sinhala Encoder-only Language Models and Evaluation and WikiBERT show that monolingual vocabularies beat multilingual models when there is enough data. WikiBERT also constrains the scope: below roughly 100M tokens, multilingual transfer wins. We therefore restrict ourselves to languages above that threshold. VocADT shows that the languages with the worst fragmentation gain most from vocabulary adaptation. That is consistent with fertility being the variable that matters.
No work in the file combines a hybrid local-global encoder with language-specific tokenization.
Research Program
We pretrain BERT-Base-sized bidirectional masked-language-model encoders in three languages chosen to span morphology and script. The languages are Slovak (inflected, Latin script), Sinhala (non-Latin abugida, SINHALA-Corpus1.5B), and Finnish (agglutinative; WikiBERT notes about 97M Wikipedia tokens, which we supplement with web text).
For each language we build a ladder of tokenizers that separates "language-specific" from "low fertility." The ladder has four rungs:
- a byte-level tokenizer;
- the shared XLM-R-style multilingual tokenizer;
- monolingual BPE at 8k and 16k vocabulary;
- monolingual BPE at 50–64k vocabulary, matching SlovakBERT and the Sinhala models.
The monolingual vocabularies of different sizes let us vary fertility inside a single language. This is how we test whether any effect comes from fertility or from vocabulary quality.
Each encoder layer combines two paths. The local path is exact bidirectional sliding-window softmax attention of $k$ tokens. The global path is LION-style bidirectional linear attention. We use separate normalizations for the two paths, following Transformer-LS. In addition, $g$ of the 12 layers are replaced by full softmax attention, placed either in the top layers or evenly spaced. We sweep $k \in \{16, 32, 64, 128, 256\}$ and $g \in \{0, 1, 2, 4\}$ at a fixed pretraining budget measured in characters, not tokens. A character budget keeps high-fertility runs from seeing less text.
Two further variants follow. The first is a fertility-scaled window: we set $k$ per language as $k = 2 w^* f$, where $w^*$ is the best word span found in the sweep. The second is a word-aligned window, in which each position attends to all tokens of a fixed number of neighboring words, whatever their token count.
The analysis has two stages. First we test curve collapse: we fit accuracy as a function of $w$ and as a function of $k$ across all language and tokenizer cells, and ask which fit explains more variance on held-out cells. Second, for each tokenizer we find the smallest $g$ that matches full attention within a fixed tolerance.
Baselines
The naive reference point is the standard recipe. It uses a hybrid with a fixed 64-token window, the BASED and LoLCATs default, trained on the shared multilingual tokenizer. The proposal claims this recipe is miscalibrated for high-fertility text, so it is the thing to beat. It shares data, character budget, and a tuning sweep of equal size with every other run.
The strongest quality competitor is a full-attention bidirectional encoder with a language-specific tokenizer, in the style of SlovakBERT and the Sinhala Raja models. We use two forms of it. The first is the published checkpoints, as an external anchor. For these, SlovakBERT reports 98.37% part-of-speech accuracy, 0.990 document-classification F1, and about 0.79 Spearman on similarity. The Sinhala work reports over 94% F1 on named-entity recognition. WikiBERT reports 86.6% average dependency-parsing labeled attachment score against mBERT's 86.1%. The second form is our own retrained full-attention encoder under the identical character budget, which makes the comparison fair. A hybrid "beats" this baseline if it comes within 0.5 points on these tasks at lower cost. It clearly beats it if the hybrid is equal or better on the long-document tasks described below at a lower FLOP count.
The strongest efficient competitor is a pure bidirectional sub-quadratic encoder with no window and no full-attention layers. We run Hydra and LION as published architectures, on the same tokenizers and budgets. No reported result was found on Lacuna for Hydra or LION on Slovak, Sinhala, or Finnish benchmarks.
The ablation that isolates the key ingredient compares three settings at matched compute: fixed $k$ = 64, fertility-scaled $k$, and word-aligned windows. If fertility scaling adds nothing over a well-tuned fixed $k$, the core claim is false.
Why not just…?
Why not just use a language-specific tokenizer and keep the 64-token window? This is the most serious alternative. Low-fertility tokenizers may already put a 64-token window in a comfortable range of word span, in which case the whole effect reduces to "tokenize well." The fertility ladder is built to settle this. If accuracy is flat in $k$ above some small value for every monolingual tokenizer, the practical recommendation collapses to "use a good tokenizer." We would then report that the window is not the bottleneck, which is itself new evidence about the BASED and LoLCATs design. We expect the effect to show up mainly at the high-fertility rungs and in Sinhala.
Why not just use full attention, since encoder inputs are only 512 tokens? At 512 tokens full attention is cheap, and the published monolingual models all work at 128–512 tokens. The argument for hybrids rests on long inputs. High-fertility tokenization makes a 512-token limit cover far less text, and document-level tasks need thousands of tokens. For this reason we report results at 512 tokens and at 4k tokens separately. If hybrids offer no advantage beyond cost at 4k either, we will say so.
Why not just use a pure bidirectional linear model like LION or Hydra? Hydra itself concedes weaker exact retrieval. BASED's 10.36-point recall margin over Mamba, and the expressiveness hierarchy's lower bounds, predict that pure linear models will fail at retrieval-heavy long-context tasks. The pure-linear baseline and the $g$ = 0 cells test this directly in the target languages.
Why not just adapt a multilingual model's vocabulary post hoc, as VocADT does? VocADT reports failure on scripts with near-zero base coverage, such as Hangul in Mistral. It was also only tested on causal full-attention decoders. Sinhala is precisely the kind of script where VocADT's authors saw it break down. We include a VocADT-adapted multilingual checkpoint as an extra tokenizer rung, and we measure its effective fertility in the same way as the others.
Proposed Experiments and Decisive Checks
Short-context evaluation uses SINHALA-GLUE (sentiment, offensive language, headline prediction, similarity, named entities, offensive tokens), the Slovak part-of-speech, similarity, sentiment, and document-classification suites, and Universal Dependencies v2.3 parsing for Slovak and Finnish.
For long contexts we build two tests from held-out text in each language. The first is document-level masked-language-model loss at 4k tokens, measured in bits per character so that results are comparable across tokenizers. The second is an associative-recall task in the style of the Multi-Query Associative Recall test, built from real named entities. We also report pretraining throughput and memory at 512 and 4k tokens.
The decisive check is the curve collapse. The claim is supported if word span $w$ explains held-out accuracy substantially better than token count $k$, and if fertility-scaled windows match or beat the best fixed window in every language while fixed 64 underperforms at high fertility. The claim is refuted if $k$ explains accuracy as well as $w$.
A second check is whether the minimal number of full-attention layers falls with fertility, and whether placing those layers at the top beats spacing them evenly. The SlovakBERT probes and the LoLCATs layer-difficulty findings predict both effects. If both hold, the paper's conclusion is concrete: window sizes should be specified in words, and a language-specific tokenizer buys cheaper hybrid encoders as well as better embeddings.
References
- Simple linear attention language models balance the recall-throughput tradeoff
- LoLCATs: On Low-Rank Linearizing of Large Language Models
- Long-Short Transformer: Efficient Transformers for Language and Vision
- Linear Attention for Efficient Bidirectional Sequence Modeling
- Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers
- DeltaFlow: Noise-Adaptive Bidirectional Gated Delta Networks for Embedded Language Flows
- A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attention
- SlovakBERT: Slovak Masked Language Model
- Sinhala Encoder-only Language Models and Evaluation
- WikiBERT models: deep transfer learning for many languages
- Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?