Reservoir Taggers with Induced Function Morphemes for Low-Resource African Languages

A fixed, randomly wired recurrent network can learn sentence structure from a few hundred examples if you first tell it which words carry grammar. Reservoir computing trains only a linear output layer, in one regression step, on top of a recurrent network whose weights are never updated. In the reservoir models of child language learning by Hinaut and colleagues, the grammatical cue is supplied by hand. Before the sentence reaches the network, every noun and verb is replaced by a placeholder token, so only function words like "the", "to" and "before" remain. Nobody has shown this works when the grammatical cues are prefixes and suffixes glued onto word roots, as in isiZulu or chiShona. Nobody has tested it on a real benchmark either.

We propose to learn that substitution instead of hand-coding it. A simple frequency rule, run on raw unlabeled text, marks a substring as a "function morpheme" if it is frequent and attaches to many different roots. The rest of each word collapses into a generic content slot. A fixed reservoir then reads the result, and linear readouts are trained per task and per language. The test is part-of-speech tagging across the typologically diverse African languages of MasakhaPOS, in the regime that matters for the digital language divide: tens to hundreds of labeled sentences and almost no raw text. The headline question is precise. Does the automated version of the child-acquisition idea "grammar lives in high-frequency closed-class forms" give a gradient-free learner a sample-efficiency advantage on real low-resource languages? And is that advantage due to the recurrent dynamics or only to the input coding?

Core Claim

The reservoir's small-data advantage in prior work comes from two ingredients that have never been separated. The first is fixed recurrent dynamics. The second is an input code that exposes closed-class structure. We claim the second can be induced without supervision for agglutinative morphology. We also claim the combination beats both fully trained recurrent taggers and windowed linear taggers when labels are very scarce, with the reservoir's specific contribution concentrated on tokens whose tag depends on context more than a few words away. We do not claim to beat a fine-tuned Africa-centric Transformer at full data. We expect the crossover to sit somewhere in the low-label regime, and locating it is part of the result.

Where This Sits

The method this proposal reuses, and the gap it targets, both come from one line of work. Exploring the acquisition and production of grammatical constructions through human-robot interaction with echo state networks maps sentences to "who did what to whom" using a fixed reservoir over function-word skeletons. It reaches over 75% generalization on 373 English robot commands, but the placeholder step is hand-coded. A Recurrent Neural Network for Multiple Language Acquisition: Starting with English and French shows one reservoir handles two languages at roughly the average of the monolingual error rates (20.6% bilingual versus 15.8% French and 21.4% English). It again relies on manual placeholders, and the two languages are related and high-resource. Recurrent Neural Networks Models for Developmental Language Acquisition: Reservoirs Outperform LSTMs and Modelling Cross-Situational Learning on Full Sentences in Few Shots with Simple RNNs find that reservoirs degrade more gracefully than LSTMs as vocabulary grows, with roughly 1,000 training examples. Both use synthetic corpora and do not compare against Transformers.

We keep their architecture and their small-data framing. We drop the hand-coded abstraction, the synthetic data, and the English-only evaluation.

Three papers constrain the design. Echo State Speech Recognition finds that randomizing a speech decoder is harmless while randomizing the encoder is severely harmful. This warns that a purely random reservoir over raw symbols may lack good features, and it is exactly why the input code, not the reservoir, carries the learned linguistic prior here. Do LLMs learn a true syntactic universal? shows large Transformers failing to acquire a word-order constraint from about 153M tokens of Basque while succeeding with German at 25.9B. That motivates looking for learners with stronger small-data biases. We do not test that constraint here and draw no conclusions about it. Constructions are Revealed in Word Distributions notes that distributional statistics alone do not fully explain how constructions are acquired. Accordingly, we treat the induced inventory as a hypothesis to be checked rather than a solution.

The application side supplies the benchmarks and the competitors. MasakhaPOS provides about 1,500 annotated sentences in each of 20 African languages, a reported 89.4% average accuracy for fine-tuned AfroXLMR-large, and evidence that transfer works best between related languages. NGLUEni documents subword fragmentation of Nguni words by shared tokenizers and supplies a Nguni corpus of about 261M tokens, which we subsample. MultiFiT shows that small recurrent models with language-specific subword vocabularies can beat multilingual Transformers cheaply. However, it trains all weights by backpropagation, which is the cost we remove.

Research Program

The pipeline has three stages: induce an inventory of function morphemes, re-code each word, and read the coded sentence with a fixed reservoir.

Stage 1: inducing the inventory. The inducer scans unlabeled text in a language, or jointly across a language family, for word-initial and word-final substrings up to a small length. It scores each candidate by two quantities: how often it occurs, and how many distinct remainders it attaches to. This is the distributional signature of closed-class forms: few types, very many tokens, promiscuous attachment. Candidates above a threshold on both scores form the inventory.

Stage 2: re-coding each word. Each word becomes a short sequence of symbols: its longest matching prefix morphemes, a content slot, and its suffix morphemes. The content slot carries a small hashed identity vector for the remainder, so that frequent roots are not entirely lost. This generalizes the hand-coded placeholder of prior work from whole words to pieces of words, and it is learned from counts alone.

Stage 3: the reservoir and readouts. A leaky echo state network reads the coded sentence forward and backward. An echo state network is a recurrent network with random weights scaled so that its state is a fading function of recent input. We also try a two-layer stack, following Deep Echo State Network (DeepESN): A Brief Survey, because upper layers integrate over longer spans. Per-token tags are predicted by ridge regression from the concatenated states at each word's last symbol.

Because the reservoir is frozen, its states are computed once per corpus. They are then reused across all label budgets, regularization values and cross-validation folds, which makes the usual tuning problem cheap (Efficient Implementations of Echo State Network Cross-Validation).

Data regimes. We vary two quantities. Labeled data runs from 50 sentences, through 100, 200 and 500, up to the full MasakhaPOS training set. The unlabeled text available to the inducer takes three settings. The most extreme uses only the unlabeled side of the training sentences themselves. The other two use 1M and 10M tokens subsampled from the NGLUEni corpus for the Nguni languages, which are the languages where we have such text.

Cross-lingual readouts. We induce one inventory jointly over the related languages present in MasakhaPOS and share one reservoir seed among them. Readouts are trained on some languages and applied, unchanged or after a few-sentence refit, to a held-out related language. This tests whether shared affix symbols let a frozen state space carry tagging knowledge across a family, as the bilingual result suggested for English and French.

Baselines

Naive reference. The floor is a most-frequent-tag-per-word tagger, with the most frequent tag overall assigned to unseen words, trained on each labeled subset. No reported result found on Lacuna for this baseline on MasakhaPOS, so we compute it ourselves.

Strongest existing method. The strongest competitor is AfroXLMR-large fine-tuned per language, as in MasakhaPOS. It reports 89.4% average token accuracy at full data. It is the right opponent because it combines Africa-centric pretraining with the same labels. For zero-shot transfer within a family we also run the MAD-X adapter setup from the same paper, which it reports as over 10 points more accurate than plain fine-tuning in zero-shot transfer.

We rerun both baselines on our exact labeled subsets, with the same development splits and a tuning budget matched in number of configurations tried. We report GPU-hours alongside accuracy. Beating this baseline means higher mean accuracy across languages at a given label budget, with confidence intervals from several subset resamples that do not overlap. We expect this only at the smallest budgets. At full data, being within a few points at a small fraction of the training compute counts as a meaningful but weaker outcome, and we will report it as exactly that.

Trained recurrent baseline. A bidirectional LSTM tagger is trained by backpropagation on the identical induced code. Its hidden size and training epochs are tuned with the same number of configurations as the reservoir. This tests the prior claim that reservoirs beat LSTMs in small data, now on real language.

Windowed linear tagger. A conditional random field or logistic regression uses the same induced morpheme symbols from a fixed window of neighboring words. It isolates whether recurrence adds anything over local context. No reported result found on Lacuna for such taggers on MasakhaPOS.

Ablations isolating the key ingredient. Two variants share everything except the input code. The first feeds the reservoir hashed whole words. The second feeds it a standard unigram subword segmentation trained on the same raw text. Together these isolate the induced function-morpheme abstraction from simply splitting words. A random projection of the window without recurrence isolates the dynamics.

Why not just…?

Why not just fine-tune AfroXLMR or attach adapters? At full labels it will very likely win, given its reported 89.4%, and we say so. Its costs are large-scale pretraining and gradient fine-tuning. NGLUEni shows it still needed adaptation on about 261M in-family tokens to stop fragmenting Nguni words, which is text most languages do not have. The learning curves settle where, if anywhere, a learner that needs counts over a few thousand sentences and one regression overtakes it. If no crossover appears even at 50 sentences, the practical claim fails, and the paper reports that.

Why not just a CRF with the same affix features? This is the most serious threat. Tagging is largely local, and good affix features may be all that is needed. The factorial design addresses it directly by crossing the induced code against plain hashed words and crossing recurrence against a fixed window. We additionally bin test tokens by distance to the nearest preceding noun. The prediction is that the reservoir's gain over the windowed model appears only in the far bins, where agreement information must be carried across words. If the windowed tagger matches the reservoir in every bin, we conclude that the acquisition story is about input coding, not dynamics, and we recast the contribution as an unsupervised morpheme inducer for linear taggers.

Why not just train a small BiLSTM or a small Transformer from scratch? The prior reservoir-versus-LSTM results come from synthetic corpora, so they may not hold for real text. Training the LSTM on the identical code with a matched tuning budget tests this on real languages. Small causal Transformers like the 125M-parameter models in the dative and SabiYarn papers were trained on 88M to 10.1B tokens. That scale is far outside our regime, and at a few hundred sentences we expect them to be weaker than the LSTM, so the LSTM is the meaningful trained comparison.

Proposed Experiments and Decisive Checks

The main experiment runs every system over the MasakhaPOS languages, the full grid of label budgets, and the unlabeled-text settings. Each cell uses five random label subsets. We report token accuracy, accuracy on out-of-vocabulary tokens, and training wall-clock time.

The first decisive check is the interaction. The induced code should improve the reservoir more than it improves the windowed tagger, and the improvement should be largest at 50 to 200 labels and on agglutinative languages such as chiShona. The second check is the distance-binned comparison described above. The third is family transfer. A readout trained on related languages, applied to a held-out one, should beat the same transfer with hashed whole-word input. The comparison point is MAD-X zero-shot accuracy measured on the same split.

We also inspect the induced inventories qualitatively, checking whether the top-ranked prefixes in the Nguni languages are recognizable class and agreement markers. Absent a resource to score against, this is a sanity check, not a metric.

The whole study is cheap. Reservoir states are computed once per language, each readout is a single linear solve, and the compute-heavy cells are the Transformer baselines.