Splitting words into their meaningful parts helps language models on rare, complex words, such as reading superbizarre as super- plus bizarre rather than superb plus noise. How much it helps, though, varies widely by task, by language, and by whether the word carries a prefix or a suffix. Today the only way to find out is to train on everything and compare. This note proposes to predict the gain from a small labeled sample by asking a question from kernel learning theory: when words are represented through their morphemes, does the target function become easier to learn, and by how much?
The tool is a long-studied failure mode of kernel methods. A kernel is a similarity function between inputs, and it defines a space of "nice" functions called a reproducing kernel Hilbert space (RKHS). Classical guarantees assume the true input-to-label mapping lies inside that space. When it does not, the problem is called misspecified. Learning still works, but more slowly, and the slowdown is governed by a single smoothness index that says how far outside the space the target sits. Our claim is that morphological decomposition is best understood as a change of kernel. Its value on a given task is how much it raises this smoothness index, and that index can be estimated from a few thousand labeled words to forecast the learning-curve exponent at full scale.
Core Claim
Let $T$ be the integral operator of a kernel over words, with eigenvalues $\mu_i$ and eigenfunctions $\phi_i$, and let $a_i = \langle f^*, \phi_i \rangle$ be the coefficients of the true label function $f^*$ (here, the conditional mean of a $\pm1$ label) in that eigenbasis. Suppose the coefficients decay as $a_i^2 \asymp i^{-\gamma}$ and the eigenvalues as $\mu_i \asymp i^{-\beta}$. Then the target lies in the interpolation space $[\mathcal{H}]^s$ for any $s < (\gamma - 1)/\beta$, because those are the $s$ for which $\sum_i \mu_i^{-s} a_i^2$ converges. Lin et al. prove that spectral algorithms reach the minimax excess risk $n^{-s\beta/(s\beta+1)}$ throughout the misspecified range $s \in (0,1]$, without assuming bounded targets, under an embedding condition that holds for many common kernels. Substituting $s\beta = \gamma - 1$ gives, to leading order,
$$\text{excess risk}(n) \asymp n^{-(\gamma-1)/\gamma},$$
so the predicted learning-curve exponent depends only on how quickly the label signal concentrates in the kernel's leading directions. The proposal is to treat $\gamma$ as a measurable property of a pair consisting of a representation and a task. We hypothesize, falsifiably, that morpheme-based kernels raise $\gamma$ relative to surface-subword and character kernels, and that the size of the rise predicts the downstream gain at full data. One caveat is that this exponent concerns squared loss, the setting where the theory is established. We report F1 alongside it and treat the mapping between the two as an empirical question.
Morphology also suggests a specific prediction. Hofmann et al. report that segmentation gains are about twice as large for prefixed words as for suffixed ones. If the claim is right, the $\gamma$ gap between morpheme and WordPiece kernels should also be larger on the prefixed subset.
Relation to Prior Work
Two bodies of work motivate this proposal without touching each other. On the morphology side, DelBERT shows that derivationally correct segmentation improves frozen-BERT classification of rare complex words on its Amazon, ArXiv and Reddit lexical datasets. Li et al. report that morphologically grounded tokens speed up early training. Batsuren et al. show that supervised canonical segmenters beat byte-pair encoding (BPE), Unigram LM and Morfessor by 30.71 absolute F1 on SIGMORPHON 2022's sentence-level task. All three measure gains after training and offer no account of when or why the gains appear. We supply such an account and test whether it predicts.
On the kernel side, several results supply the method. Lin et al. and Dieuleveut et al. give minimax rates under misspecification, the former for spectral algorithms and the latter for averaged stochastic gradient descent. Li et al. show that the Nyström low-rank approximation keeps those rates, which makes the 240k-word Amazon set tractable. Smale et al. argue that targets usually fall outside the RKHS, which is our starting premise. None of these papers estimates the smoothness index on real data or uses it to compare input representations. They validate rates only on synthetic Sobolev targets or splines.
String kernels over character n-grams and gapped k-mers (GaKCo) and random-feature string kernels (global string kernels) are the closest existing kernel treatment of words. They never use morpheme boundaries and never analyze misspecification. We use one of them as the character baseline.
One result constrains the design, and one tension shapes the secondary question. Dicker et al. show that kernel ridge regression (KRR) saturates on smooth targets while kernel principal component regression (KPCR) does not, with simulated mean squared error of 0.0034 versus 0.0003. Lin et al., by contrast, find that KRR is minimax optimal for rough targets. The estimate of $\gamma$ tells us which regime a linguistic task is in. This bears directly on probing work such as Ács et al., which relies on ridge-type probes.
Research Program
We first build three kernels over the same word lists, all using the same frozen encoder (BERT-base) so that only the decomposition changes. The surface kernel is a Gaussian kernel on mean-pooled WordPiece embeddings. The morpheme kernel is additive, $k = k_{\text{root}} + k_{\text{affix}} + \lambda\, k_{\text{root}} k_{\text{affix}}$. Here $k_{\text{root}}$ and $k_{\text{affix}}$ are kernels on the embeddings of the root and the affix string obtained from a segmenter, and the weight $\lambda$ on their product controls how much non-additive interaction is allowed. For English we use DelBERT's derivational segmentation. For the multilingual arm we use a SIGMORPHON-trained canonical segmenter. The character kernel is a gapped k-mer string kernel.
Second, we estimate $\gamma$ from a labeled sample of 1k to 2k words. We eigendecompose the Nyström-approximated Gram matrix, project the labels onto the empirical eigenvectors, and fit the decay of the squared projections. The fit uses only the head of the spectrum, above the noise floor of roughly $\sigma^2/n$ (label noise variance over sample size), where projections stop carrying signal. The main technical risk sits here: empirical eigenvectors at small $n$ may misrepresent the population spectrum. We will therefore report bootstrap intervals and check stability as the sample size doubles.
Third, we fit KRR, KPCR and averaged unregularized least-mean-squares stochastic gradient descent (single-pass SGD on squared loss, the algorithm of Dieuleveut et al.) on nested subsets up to the full data. From these we read off the empirical learning-curve exponents and F1.
Finally, we fit the interaction-free morpheme kernel ($\lambda = 0$) and inspect the words with the largest residuals. If the theory is right, these are where misspecification localizes. We expect them to be semantically opaque derivations, and we will hand-annotate a sample of 300 to test that.
Baselines
The naive reference point is a majority-class predictor, together with the surface-kernel model, which represents doing nothing about morphology. For the downstream comparison, the strongest existing method is DelBERT. On its lexical datasets Hofmann et al. report F1 of .639 on Amazon, .723 on ArXiv and .701 on Reddit, against .624, .700 and .664 for WordPiece BERT. We will rerun both on our splits with the same frozen encoder and the same tuning budget for the classifier on top, because our kernel heads differ from their linear probe. The morpheme-kernel model beats DelBERT if it matches or exceeds DelBERT's F1 on our splits while also yielding a forecast that DelBERT does not provide. Accuracy parity is enough here, since the contribution is the forecast.
For the forecasting claim, the strongest competitor is empirical power-law extrapolation. This fits $a n^{-b} + c$ to error measured on the same small subsets and extrapolates. It gets identical labeled data and identical subset sizes. We beat it if the spectral exponent $(\gamma-1)/\gamma$ predicts the observed full-data exponent, and the ranking of representations by full-data F1, with lower error across the three datasets and the multilingual tasks.
Two further comparisons isolate key ingredients. To isolate linguistic boundaries, we run the morpheme kernel on random segmentations with the same number of pieces per word, and separately on BPE segmentations. If these raise $\gamma$ as much as true morphemes, the effect comes from splitting, not from morphology. To isolate the choice of regularizer, we compare KRR against KPCR at matched cross-validation budgets.
Why not just…?
Why not just fit power laws to small-sample learning curves? This is the real threat, and it is a baseline above for that reason. Power-law extrapolation needs several training runs at increasing $n$, and at small $n$ the exponent is notoriously confounded by the noise floor $c$. The spectral estimate uses one eigendecomposition and separates representation effects (the eigenvalue decay $\beta$, computable without labels) from target alignment ($\gamma$). If extrapolation forecasts as well, we will report that the spectral view adds interpretation but not prediction, and redirect toward the diagnostic use: which words carry the misspecification.
Why not just cross-validate segmenters at full scale? For English with 240k labeled words, that works, and we do it as ground truth. The forecast matters where full-scale labels do not exist, in low-resource morphologically rich languages of the kind Silva Plata discusses. The multilingual arm tests whether a forecast from 2k words transfers across languages.
Why not just fine-tune a larger model and let scale absorb tokenization? DelBERT's gains persist at BERT scale, which is evidence that scale does not simply remove the problem. Our question is also not "what is the best classifier" but "what does a representation buy". Fine-tuning would change the representation and the head together, which confounds exactly the quantity we want to measure.
Proposed Experiments
The English arm uses the three DelBERT lexical datasets, split by prefixed versus suffixed words. The multilingual arm uses Universal Dependencies morphological-tag tasks from Ács et al. for Czech, Hungarian and Russian, languages that SIGMORPHON 2022 also covers. These are inflectional targets. We predict a high $\gamma$ under the affix kernel, near the attainable case, unlike the derivational sentiment and topic targets.
The claim survives only if four checks hold. First, the exponent predicted from at most 2k words falls within the bootstrap interval of the exponent observed from 2k to full data on most dataset-representation pairs, and does so more often than power-law extrapolation. Second, the ordering of the three kernels by $\gamma$ matches their ordering by full-data F1. Third, the $\gamma$ gap between morpheme and surface kernels is larger on prefixed than on suffixed words, mirroring DelBERT's twofold difference. Fourth, random-boundary segmentation does not reproduce the gain.
As a secondary result, we will report whether KPCR beats KRR on the high-$\gamma$ inflectional tasks and not on the derivational ones. That outcome would resolve the Dicker-versus-Lin tension for linguistic data. If none of the four checks holds, we conclude that the smoothness index is not a useful lens for tokenization, which is itself a clean negative result.
References
- On the Optimality of Misspecified Spectral Algorithms
- Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words
- When is Char Better Than Subword: A Systematic Study of Segmentation Algorithms for Neural Machine Translation
- The SIGMORPHON 2022 Shared Task on Morpheme Segmentation
- Nonparametric stochastic approximation with large step-sizes
- Optimal Convergence Rates for Agnostic Nyström Kernel Learning
- Shannon sampling and function reconstruction from point values
- GaKCo: A Fast Gapped k-mer String Kernel Using Counting
- Efficient Global String Kernel with Random Features: Beyond Counting Substructures
- Kernel ridge vs. principal component regression: Minimax bounds and the qualification of regularization operators
- Morphosyntactic probing of multilingual BERT models
- Tokenization as Cultural Erasure: How Corpus Composition Shapes the Representation of Aymara Morphology in NLP Systems