Some language models for morphologically rich languages do not read raw subword fragments. They first run a morphological analyzer, a rule-based or statistical tool that splits each word into a stem, prefixes, suffixes and grammatical tags, and then encode those parts. KinyaBERT for Kinyarwanda is the clearest example. These analyzers fail on names, loanwords, slang, typos and invented words. When they do, the model gets empty morphological slots. The same empty slot can mean two different things: the word really has no affix in that position, which linguists call a null or zero morpheme and which carries grammatical meaning, or the analyzer could not tell. Current designs cannot distinguish the two. This is the "placeholder ambiguity" that the missing-data literature has already diagnosed and repaired for tabular and sensor inputs.
We propose to give each morphological slot of each word an explicit three-way state (overt morpheme, confidently null, unobserved). A small gate, conditioned on that state pattern, decides per word how much to trust the morphology stream and how much to trust an always-available subword stream. The claim is testable and narrow. The gain should appear where analyzer failure is informative about the label, for example named entities in NER, and it should grow when failures are concentrated on particular word types rather than scattered at random. If the gain appears only under random failures, or disappears once null and unobserved are merged into one state, the idea is wrong.
Core Claim
In the missing-data literature, Le Morvan et al. (2020) show that optimal prediction with missing inputs needs a different sub-model for each pattern of which values are missing. They also show that a neural network given a binary missingness mask alongside the inputs can learn those pattern-dependent switches, and that it stays consistent even when missingness depends on the missing values themselves. A related line of work shows that imputing first and then predicting throws this signal away (Le Morvan and Varoquaux, 2025; Perez-Lebel et al., 2022). Explicit indicators also resolve the ambiguity between a real zero and a placeholder zero (Gauch et al., 2025). These results motivate the method but were proved only for fixed-length real-valued feature vectors.
A word's morphological analysis is exactly such a partially observed, fixed-width record: $K$ slots, some filled, some empty. It also has a distinction tabular data rarely has. An empty slot can be a true observation of absence. So we ask whether pattern-conditioned switching carries over to discrete word structure. We also ask whether separating "null" from "unobserved" is what does the work.
Research Program
We build on the two-tier design of KinyaBERT. That model's first tier encodes an analyzer's stems, affixes and agreement tags into one vector per word, and its second tier is a sentence Transformer. Its reported gains over subword baselines assume analyses are available. The file gives no account of how it treats unanalyzable words, so we treat that as an open design choice rather than a known weakness.
Our word encoder has two streams. The morphology stream embeds each slot's content. The subword stream pools the word's ordinary subword tokens from the unchanged statistical tokenizer, which keeps its compression. Each word carries a state vector $s \in \{\text{overt}, \text{null}, \text{unobserved}\}^K$, where $K$ is the number of morphological slots. The word representation is
$$h_w = g(s) \odot f_{\text{morph}}(\text{slots}, s) + (1 - g(s)) \odot f_{\text{sub}}(\text{subwords}),$$
where $f_{\text{morph}}$ and $f_{\text{sub}}$ are the two stream encoders, $g(s) \in [0,1]$ is a per-dimension gate computed by a small ReLU network from the state vector alone, and $\odot$ is elementwise product. The null and unobserved states get separate learned embeddings inside $f_{\text{morph}}$. The analyzer's own signals decide which state a slot takes. A complete parse with an empty slot is null. A failed or low-confidence parse is unobserved, either for that slot or for the whole word, which also covers partial parses such as a recognized prefix on an unknown stem.
Pre-training follows KinyaBERT's joint stem-and-affix masked prediction. Loss on unobserved slots is skipped rather than imputed, which follows the Direction 1 finding that imputation separates completion from the task loss.
The analysis part of the program measures how informative absence is before any model is trained. On each benchmark we estimate the mutual information between the analyzer's failure pattern and the gold label, such as entity versus non-entity tokens. Direction 1 theory predicts that indicator gains scale with this dependence. So we pre-register the prediction that per-language gains will rank-correlate with it.
Relation to Prior Work
Three lines of work respond to subword failures by changing what the model reads. Character-level encoding is one: CANINE hashes codepoints and downsamples, and it beats mBERT on TyDi QA. Monolingual vocabularies are another: Mekonnen et al. (2025) cut Amharic fertility (tokens per word) to 1.46. Explicit morphology is the third, through KinyaBERT or compound splitting as in Huck et al.. None of them represents the reliability of the structure they supply. We keep the subword input and the morphology input and add only that reliability signal. Truong et al. (2024) show that forcing affix boundaries with hyphens lifts Llama-2 negation accuracy from about 77% to about 96%, but nonce words remain hard. That is evidence the missing piece is structure the tokenizer hides. It is also the regime, partial parses of invented words, that our unobserved state targets. Jin et al. (2020) fill missing paradigm cells by generating strings. We instead leave cells missing and flag them, which the Direction 1 results predict is better for downstream prediction.
The main constraint comes from Evaluating Morphological Alignment of Tokenizers in 70 Languages. It finds that tokenizer–morpheme alignment explains almost none of the downstream variance (recall $R^2 = 0.024$). We take this seriously. It is why we do not propose a more morphological tokenizer and why compression is left intact. It also means that a null result from us would be consistent with that paper, and we would report it as such.
Baselines
The naive reference is a standard subword encoder of the same size, pre-trained on the same Kinyarwanda corpus with no morphology input. It shows whether morphology helps at all under our compute budget. For MasakhaNER the external reference is XLM-R, which Adelani et al. (2021) report at average F1 "around 80-83%" across the ten languages. No per-language Kinyarwanda F1 appears in the file. We will therefore re-run XLM-R-base fine-tuning ourselves on the same splits.
The strongest existing method is a faithful KinyaBERT reimplementation, trained on identical data, tokens seen and tuning budget. Unanalyzable words fall back to a single shared placeholder embedding. The file reports KinyaBERT beating baselines by "2-3.2% on NER" and "4.3% on GLUE" (Nzeyimana and Rubungo, 2022), but no absolute MasakhaNER Kinyarwanda score. Beating it means a gain over our reimplementation larger than the seed-to-seed spread across five seeds, concentrated on tokens the analyzer fails on.
A pre-segmentation baseline feeds analyzer boundaries into the text as markers, as with the hyphenation trick or German compound splitting. Failed words are left unsegmented, and a standard subword model reads the result. An imputation baseline fills unobserved slots with the pre-trained model's own most likely slot prediction before encoding. CANINE-C at matched parameters covers the tokenizer-free option.
Two ablations isolate the key ingredient. The first merges null and unobserved into one state. The second keeps all three states but freezes the gate at a constant. Every system gets the same pre-training corpus, the same number of pre-training steps, and the same hyperparameter search of twenty trials per task.
Why not just…?
Why not just use a learned "unknown" embedding for failed slots? In embedding-based models a dedicated missing token already acts as a missingness indicator, so this is the most serious threat. It is exactly our merged-state ablation. If it matches the three-state model, the contribution reduces to "KinyaBERT with an UNK slot," and we will say so. Our bet is that merging forces one vector to mean both "grammatically unmarked" and "no information." We expect this to hurt most on agreement-sensitive GLUE-style tasks and on partially parsed nonce words.
Why not just pre-segment with the analyzer and use an ordinary subword model? Truong et al. (2024) show that surface boundary insertion works well when boundaries are known. It has no way to mark a zero morpheme and no signal for a failed parse, because the failed word simply looks like any other unsegmented word. The pre-segmentation baseline tests this directly. If it matches us on in-domain text, our advantage should still show up in the controlled-degradation experiment below.
Why not just go tokenizer-free? CANINE outperformed mBERT by 2.8 F1 on TyDi QA passage selection (Clark et al., 2021). But it gives up substring priors that Peters and Martins (2022) found useful for morpheme segmentation, and it pays for sequences several times longer. We include it at matched parameters and report both accuracy and training cost.
Why not just impute the missing analysis? Le Morvan and Varoquaux (2025) show that imputation discards informative absence. The imputation baseline tests whether this holds for words. It should lose specifically on entity tokens, where failure itself is the signal.
Proposed Experiments
The main evaluation uses the KinyaBERT suite (translated GLUE, MasakhaNER Kinyarwanda, and NEWS categorization), with F1 and accuracy broken down by analyzer state: fully parsed, partially parsed, and failed words.
The decisive check is controlled analyzer degradation. On the Kinyarwanda training and test data we hide analyses at rates from 0% to 60% under two regimes. In the random regime, words are hidden uniformly. In the informative regime, analyses are hidden preferentially for rare and capitalized words, which mimics real failures. Direction 1 theory predicts that the three-state gated model's lead over the placeholder and imputation baselines will widen with the hiding rate in the informative regime and stay small in the random regime. A flat or reversed pattern falsifies the proposal's mechanism.
A second check uses the affixal negation probes of Truong et al. with an English morphological analyzer. Nonce words like dis-roagly yield a recognized prefix and an unobserved stem. We will test whether the partial-parse representation recovers negation where the pre-segmentation baseline cannot.
Finally, we extend to other MasakhaNER languages for which an analyzer is available. There we test the pre-registered prediction that per-language gains rank-correlate with the measured dependence between analyzer failure and entity labels.
References
- Linear predictor on linearly-generated data with missing values: non consistency and solutions
- Imputation for prediction: beware of diminishing returns.
- Benchmarking missing-values approaches for predictive models on health databases
- How to deal w___ missing input data
- KinyaBERT: a Morphology-aware Kinyarwanda Language Model
- Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
- Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval
- LMU Munich's Neural Machine Translation Systems for News Articles and Health Information Texts
- Revisiting subword tokenization: A case study on affixal negation in large language models
- Unsupervised Morphological Paradigm Completion
- Evaluating Morphological Alignment of Tokenizers in 70 Languages
- MasakhaNER: Named Entity Recognition for African Languages
- Beyond Characters: Subword-level Morpheme Segmentation