Pixel language models avoid the vocabulary problem altogether. They render text as an image and read it with a vision transformer, so a new alphabet, an unusual diacritic, or a typo never becomes an "unknown token." That property matters most for low-resource languages, whose scripts are poorly covered by shared subword vocabularies. The cost is sequence length. Every 16×16 pixel square of rendered text becomes a token, and attention cost grows with the square of the token count. A line of sparse Latin text and a line of dense Amharic or Tamil, full of stacked vowel signs and ligatures, are cut into the same uniform grid. Many squares are wasted on blank space and simple strokes, while the dense ones are packed with meaning-bearing detail.
High-resolution computer vision has faced the same mismatch between what is uniform in pixels and what is uniform in information, and it has a standard remedy. A cheap low-resolution pass decides where fine detail lives, and full-resolution computation is spent only there. We propose bringing this remedy to pixel language models. A small network looks at a coarse version of the rendered line and predicts, region by region, how much visual detail the text carries. Low-detail regions are merged into wide, cheap tokens. High-detail regions keep fine patches. Under a fixed total token budget, the model then spends more of that budget on complex scripts and less on whitespace and simple glyphs. The bet is that this cuts the memory and attention cost of multilingual pixel models by roughly half on low-resource benchmarks without losing accuracy. A fixed rendering scheme or a fixed compression budget cannot do this, because neither can tell a dense line from a sparse one.
Core Claim
The claim is narrow and testable. At an equal token budget, allocating patches according to a learned measure of local visual detail beats three cheaper strategies on low-resource tasks, and the margin is largest on scripts with high grapheme density. The three cheaper strategies are uniform patching, fixed-width character rendering, and simply discarding blank patches. The detail measure comes from the model's own pretraining signal and needs no labels: we use how hard each patch is to reconstruct under masked image modeling.
If dropping whitespace patches alone matches the learned allocation, then the "adaptive" part adds nothing and the simpler method should be adopted. We design the experiments so this outcome is visible.
Where This Sits
The proposal builds directly on three pixel-model papers.
Language Modelling with Pixels introduced PIXEL, which renders text into 16-pixel-high strips and pretrains a vision transformer by masking and reconstructing patches. It showed strong zero-shot transfer to unseen scripts (about 96% POS accuracy on Coptic versus about 26% for BERT) and robustness to visually confusable characters. It also noted that its long patch sequences are expensive.
Multilingual Pretraining for Pixel Language Models pretrained the same architecture on English, Hindi, Ukrainian, and Chinese. Its large SIB-200 gains on non-Latin scripts motivate our work, because they show multilingual pixel pretraining is worth scaling. It left patch-sequence growth and scaling to many more scripts open, and that is the constraint we target.
Text Rendering Strategies for Pixel Language Models is the closest prior attempt to make patches more efficient. Its BIGRAMS strategy renders exactly two characters per patch, which reduces visual redundancy and improves MasakhaNER by 3.6 points on average. However, it fixes the information per patch by character count. That assumption breaks for Brahmic conjuncts and connected cursive scripts, where "two characters" can mean very different amounts of ink. Our allocation is driven by measured visual detail, not by character boundaries.
The mechanism comes from high-resolution vision. Ultrahigh Resolution Image/Video Matting with Spatio-Temporal Sparsity uses a 512p prior network to gate sparse high-resolution computation onto boundary bands, cutting video compute by over 92%. Attention-Driven Cropping for Very High Resolution Facial Landmark Detection uses a coarse pass to choose crops from a full 4K canvas. Both papers also constrain our design. They report that regions the coarse stage misses cannot be recovered. For that reason we never delete low-detail regions; we merge them into coarse tokens that stay in the sequence.
mPLUG-DocOwl2 compresses document pages to a fixed 324 tokens using cross-attention from low- to high-resolution features. It is evaluated only on English and other high-resource benchmarks. Its fixed per-page budget is exactly the design we argue against for scripts whose information density varies.
Research Program
Each text line is rendered at 16 pixels high, as in PIXEL-M4. A scorer, a three-layer convolutional network with under 1M parameters, reads a 4× downsampled copy of the line and outputs one detail score per 16-pixel column.
A merging step then walks along the line. It merges adjacent low-scoring columns into a single token of width 32 or 64 pixels, using average pooling followed by the shared patch projection. High-scoring columns stay as 16×16 patches. A target ratio $\rho$, the fraction of uniform tokens kept, sets the threshold for each batch. Every token carries its start offset and width as positional information, so the encoder knows how much horizontal span each token covers.
The scorer is trained without labels. Its target for each column is the masked-reconstruction loss that a uniform-patch PIXEL-M4 checkpoint incurs on that column. We regress the score onto this target and fine-tune it jointly during continued pretraining. Masked image modeling then operates on the variable-width tokens. The decoder reconstructs pixels for the full span of each masked token, so coarse tokens are still trained to hold their content.
We pretrain on the PIXEL-M4 languages drawn from mC4, matching its steps and data. As a second phase, we spend the saved budget on adding Ge'ez (Amharic) and Tamil scripts. This tests whether the efficiency gain can be converted into broader script coverage.
The analysis asks where the budget goes. We measure tokens per character by script. We also check whether high-scoring columns align with diacritics, conjuncts, and dense Han characters, using rendered glyph-complexity counts as a reference.
Baselines
The naive reference point is uniform-patch PIXEL-M4 at full sequence length, run on identical pretraining data, steps, and fine-tuning budget. It defines the accuracy ceiling we hope to keep while using about half its tokens.
A second naive point is uniform downsampling, implemented as rendering at 8-pixel height so the line produces half as many tokens. It shows what happens when the budget is cut without regard to content.
The strongest pixel-side competitor is BIGRAMS rendering from Text Rendering Strategies for Pixel Language Models. It is the only published method that reduces patch redundancy, and it reports a 3.6-point average MasakhaNER gain over continuous rendering. We pretrain it multilingually under the same compute so that it matches our setup.
The key ablation is whitespace pruning: drop or merge only blank columns, keep every inked column at full resolution, and leave everything else identical. It isolates whether learned detail scores matter beyond the obvious step of removing empty space. A second ablation replaces the learned scorer with a hand-made ink-density score, which tests whether reconstruction difficulty is a better signal than pixel counting.
On the text side, the references are XLM-R and XLM-V. For XLM-R on MasakhaNER, MasakhaNER reports average F1 of roughly 80–83. For XLM-V, XLM-V reports +11.2 absolute F1 over XLM-R on MasakhaNER.
For PIXEL-M4 itself, the file gives only gains relative to English-only PIXEL-BIGRAMS on SIB-200 (+46.0 F1 in Hindi, +36.1 in Ukrainian). We found no reported absolute PIXEL-M4 SIB-200 or MasakhaNER score on Lacuna, so we will reproduce it ourselves.
Beating the baselines means one of two things. The first is staying within 1 F1 of full-sequence PIXEL-M4 at $\rho = 0.5$ while clearly exceeding whitespace pruning and BIGRAMS at that same ratio. The second is outperforming full PIXEL-M4 when both use the same peak memory, with the saved memory spent on more context or the added scripts.
Why not just…?
Why not just drop the blank patches? This is the most serious threat. On Latin text much of the redundancy is whitespace, and pruning it is trivial. It cannot, however, reach a 50% budget on dense scripts. It also treats a column holding one plain stroke the same as one holding a stacked conjunct. The whitespace-pruning ablation, run at each target ratio and broken down by script, settles the question. If it matches the learned scorer on Amharic and Tamil, we report that and recommend it.
Why not just use fixed-width character rendering like BIGRAMS? BIGRAMS works well and is cheap. But it allocates by character count, and the authors' own limitation is that this breaks for conjuncts and cursive scripts. The per-script comparison at equal token counts will show whether that limitation actually costs accuracy on Devanagari, Ge'ez, and Tamil.
Why not just compress with a fixed cross-attention budget like DocOwl2? A fixed budget per unit of input is exactly what we hypothesize is wrong for variable-density scripts, and mPLUG-DocOwl2 was never tested on such scripts. We include a learned-query pooling variant with a fixed output length per line as an additional ablation. If it matches adaptive merging, the claim about adaptivity fails.
Why not just use a bigger discrete vocabulary? XLM-V reports that gains stop beyond about 1M tokens and that the 1M vocabulary multiplies parameters by 3.3. A vocabulary also remains closed to unseen scripts and orthographic noise. We report XLM-V as a reference and evaluate robustness under visual character substitution, where pixel models are expected to keep their advantage.
Proposed Experiments
Evaluation covers the following tasks. MasakhaNER covers ten African languages, including Amharic in Ge'ez script. SIB-200 provides topic classification across many scripts. Universal Dependencies POS tagging includes Coptic and the other non-Latin treebanks used by PIXEL. The visually confusable character attacks from PIXEL test robustness. We add Tamil and Malayalam hope-speech detection from LT-EDI as a noisy social-media setting. For that task, TeamUNCC reports weighted F1 of 0.61 on Tamil and 0.83 on Malayalam with XLM-R.
For each method we report task score against tokens per line, peak GPU memory, and fine-tuning throughput, sweeping $\rho \in \{1.0, 0.75, 0.5, 0.35\}$. This yields accuracy–memory curves rather than single points.
Three checks decide the outcome. The first is whether, at $\rho = 0.5$, the adaptive model's loss relative to full PIXEL-M4 is smaller than whitespace pruning's loss on the three densest scripts. The second is whether the scorer's allocation correlates with glyph complexity across scripts, rather than simply tracking ink. The third is whether, at equal peak memory, spending the savings on Ge'ez and Tamil pretraining improves Amharic MasakhaNER and Tamil hope-speech over full-sequence PIXEL-M4.
A negative answer on the first check would mean that content-adaptive patching is not worth its complexity for pixel language models, which is itself a useful result.
References
- Language Modelling with Pixels
- Multilingual Pretraining for Pixel Language Models
- Text Rendering Strategies for Pixel Language Models
- Ultrahigh Resolution Image/Video Matting with Spatio-Temporal Sparsity
- Attention-Driven Cropping for Very High Resolution Facial Landmark Detection
- mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding
- MasakhaNER: Named Entity Recognition for African Languages
- XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models
- TeamUNCC@LT-EDI-EACL2021: Hope Speech Detection using Transfer Learning with Transformers