Inflection Rules as Tested Programs: Executable Role Decomposition Against the Real-Word Bias

Large language models are surprisingly bad at a task that children handle easily: inflecting a word they have never seen. Given a made-up stem and a few examples of a grammatical pattern, a chat model often writes a real word that looks similar instead of applying the pattern. For Tamil it can fail completely. We propose that the model should never write the inflected form at all. One model role studies the examples and segments them into parts, a second writes the inflection rule as a small executable program, and a third tests that program against held-back examples and contrastive probes before it is run on the new stem. The output is whatever the program produces. Because no model is ever asked to "say the word," the pull toward familiar real words has no channel into the answer.

The opportunity rests on two findings. In the first, chat models fail the multilingual nonce-word inflection test (the "wug test") in a specific way. In the second, role-specialized agent teams in code generation gain most where a dedicated tester finds edge cases. Morphology fits the code setting unusually well: an inflectional pattern is a string transformation, and the paradigm examples are its unit tests. If the approach works, it would give low-resource languages something they currently lack, namely a way to obtain a usable, inspectable inflection module from a handful of examples. Building such a module by hand takes hours of expert effort, and training a neural one takes many GPU-hours.

Core Claim

The paper that motivates this work is Counting the Bugs in ChatGPT's Wugs, which tested a single prompted model, gpt-3.5-turbo-0613, on nonce-word inflection in English, German, Tamil and Turkish. The model reached about 78% in English against about 98% for supervised baselines, about 59% in Turkish against about 96%, and at best about 43% in Tamil against about 85%, with 0% in zero-shot and one-shot long prompts. The authors trace many errors to a "real word bias": asked to inflect dedo, the model produces did. They only tried single-prompt variants.

Our claim has two parts, and we test them separately. The primary claim is about execution grounding. If the surface form comes from running a model-written rule rather than from model-written text, the real-word errors will mostly disappear. Accuracy on Tamil and Turkish nonce words should rise substantially above the monolithic prompt with the same model and the same examples. The secondary claim is about role separation. A tester role that is kept separate from the rule writer, and that holds out part of the examples and writes its own contrastive probes, will catch overfit rules that a single model checking its own program misses. We frame the second claim as a hypothesis that could fail, because a single model running its own code is a strong competitor.

The closest structural precedent is Self-collaboration Code Generation via ChatGPT. There, an analyst, a coder and a tester share a common workspace, improving HumanEval Pass@1 by 29.9% and edge-case variants by up to 34.6%. We borrow that method directly. The difference is that our "requirements" are a few inflected pairs, and success is judged on held-out lexical items rather than on given test cases. The task is therefore inductive generalization, not specification satisfaction.

Research Program

The system has three roles over a shared workspace that records each role's outputs.

The analyst receives the k example pairs (lemma plus feature bundle mapped to an inflected form). It proposes a segmentation of each pair into stem and affix material, and it notes the conditioning properties it thinks matter, such as final vowel, vowel class or syllable count. Vowel harmony is the main case, where the suffix vowel copies a property of the stem vowels. Rule-based Machine translation from English to Finnish singles out this kind of stem-conditioned suffix choice as the part of realization that blind concatenation gets wrong.

The coder turns the analysis into a short Python function. The function maps a stem and a feature bundle to a surface string, built from conditional string edits. This deliberately resembles the edit-script classes in Broad Coverage Multilingual Deep Sentence Generation with a Stochastic Multi-Level Realizer, which treated inflection as choosing among string edits and reached 97.8% morphological accuracy in English. We use the same output space, but the edits are induced from a few examples by a language model rather than learned by a classifier from a treebank.

The tester never sees the coder's reasoning. It splits the examples, giving the coder a fit set while keeping a check set hidden from it. It runs the function on the check set. It also constructs contrastive probes: invented stems that differ from the examples in exactly one conditioning property. It then asks the analyst, not the coder, whether the program's output on each probe agrees with the analyst's stated conditioning claims. Failures go back to the coder, or to the analyst when the segmentation itself looks wrong. This back-channel addresses error cascades, which A survey on LLM-based multi-agent systems names as a primary failure mode: downstream roles otherwise accept a bad upstream specification as ground truth.

The loop stops when the check set passes or a fixed budget runs out. The final program is applied to the nonce stem. When several programs survive, they vote by executed output.

The design also borrows from Radical Allomorphy, which avoids abstract underlying forms and instead selects among observed surface allomorphs using surface phonotactic scores. It reached 85.2% on unseen Egyptian Arabic words versus 81.6% for the rule inducer PARLA, and 52.6% versus 4.1% for a neural Transformer at 100 training samples. We give the tester a character n-gram phonotactic scorer trained only on the examples and any available unannotated word list, used as one extra check. This paper therefore plays two roles for us. It supplies a method we reuse, and it constrains us as a strong non-agentic competitor we must beat.

Beyond the wug test, we target one application regime: replacing the hand-written inflection module in terminology translation. Rule-based Morphological Inflection Improves Neural Terminology Translation showed that a rule-based inflector driven by dependency-derived features reached 77.4% on Lithuanian versus 15.4% for a neural inflector. The rules took 3–6 human hours to build, against 38 GPU-hours for the neural model. If induced programs approach the hand rules, the human effort that paper relied on can be automated.

One constraint shapes the model choices. Dong et al. report that models under roughly 13B parameters fail to maintain roles. We expect that passing executable functions between roles, with mechanical checks, depends less on maintaining a persona than free-text role-play does. We test this with open models at 8B and 70B scale alongside a closed frontier model.

Baselines

The naive reference point is the monolithic prompt from Weissweiler et al., rerun by us with the same models used in our system. The originally reported gpt-3.5-turbo-0613 figures are about 78% (English few-shot), about 59% (Turkish), and at best about 43% (Tamil), with 0% for zero-shot and one-shot long prompts. That model generation may no longer be the right comparison, so the comparison that matters is our own rerun with identical examples. We also report a self-consistency variant of the monolithic prompt, which samples many answers and takes the majority. We give it the same total token budget as the full three-role system, so any gain cannot be attributed to more inference compute.

The strongest existing method on these exact items is the supervised baseline reported by Weissweiler et al.: about 98% in English, about 96% in Turkish and about 85% in Tamil. These systems see far more training data than our k examples. We do not expect to beat them at full data. We will count it as a win if the program-based system closes most of the gap, defined as more than half the distance between the monolithic prompt and the supervised score in Tamil and Turkish, while using only the in-context examples.

The fairer head-to-head is a data-matched supervised system. We will train the Transformer and the radical-allomorphy selector on exactly the k examples the agents see, with equal hyperparameter search. On Egyptian Arabic, at 100 samples, the reported bar is 52.6% for radical allomorphy and 4.1% for the Transformer. Beating 52.6% at matched data is our threshold for claiming the method helps low-resource realization.

For Lithuanian terminology inflection the bar is the hand-built rule module's 77.4%. For SIGMORPHON 2020 low-resource languages under the UniMorph schema (SIGMORPHON 2020 Shared Task 0), we found no reported result on Lacuna for per-language accuracy at our subsampled sizes. We will therefore train and report the shared-task-style neural baseline ourselves on the same subsamples.

Two ablations isolate the key ingredients. The first is execution grounding: the same three roles, but the coder writes the inflected form directly instead of a program. The second is role separation: a single model writes the program, runs it on all the examples, and repairs it on failure, with the same budget and no hidden check set or contrastive probes.

Why not just…?

Why not just prompt a stronger model with chain-of-thought? Weissweiler et al. show the failure persists under few-shot prompting and is systematic rather than random: errors gravitate toward real words. Chain-of-thought still ends with the model writing the form, which is the channel the bias acts through. We do not argue this from the literature alone. The monolithic and self-consistency baselines run on the same frontier model, and we report the share of errors that are real dictionary words. Our hypothesis predicts that this share falls sharply with execution and stays high without it. If a stronger model with chain-of-thought matches the program-based system, the primary claim fails for that model, and we will say so.

Why not just have one model write and run its own program? This is the real threat. It is essentially program-of-thought with execution feedback, and it may capture most of the gain. The code-generation evidence from Dong et al. favors a separate tester, but there the tester probes given requirements. Here the risk is a program that fits every example yet encodes the wrong conditioning factor, and a self-checking model that sees all examples cannot detect that. The role-separation ablation settles this directly. If it matches the full system within noise, we will report execution grounding as the contribution and drop the multi-agent claim.

Why not just generate candidates and rank them with a phonotactic scorer, with no agents? Radical allomorphy shows that discriminative selection over surface candidates is strong in low-resource settings. A sampler-plus-scorer pipeline is cheap. Its weakness is that a character n-gram model trained on a few examples cannot tell a well-formed real word from the correct nonce inflection. That is exactly the bias we target, and the original paper relied on a pre-enumerated set of allomorphs rather than an open-ended proposer. We run this pipeline as a baseline using the same candidate pool and the same scorer we give the tester.

Why not just train a small supervised inflector? At matched data, the reported neural numbers (15.4% Lithuanian, 4.1% Egyptian Arabic at 100 samples) suggest it will lose, but we rerun it on our exact splits rather than assume so.

Proposed Experiments

The main experiment uses the multilingual wug items of Weissweiler et al. in all four languages, with k ∈ {1, 4, 16} examples per pattern, and reports accuracy against their gold annotations. For English we also report agreement with the human answer distribution, because human judges disagree on some nonce forms. The second experiment is the Egyptian Arabic out-of-vocabulary benchmark of Khalifa et al. at 100 training samples, where agents are given examples retrieved from those samples. The third subsamples low-resource UniMorph languages from SIGMORPHON 2020 to 100 pairs. The fourth plugs the induced Lithuanian program into the terminology pipeline of Xu and Carpuat and measures both inflection accuracy and terminology usage accuracy.

Across all experiments we log tokens and wall-clock time per item, against the overhead concern raised in the agent survey. We also log cascade events, meaning cases where an analyst error survived to the final output, so that the back-channel's value is measured rather than assumed. Every condition runs at 8B, 70B and frontier scale, which tests whether the reported parameter threshold for role-play applies to executable interfaces.

Three checks decide the outcome. First, does execution grounding cut the real-word share of errors and lift Tamil and Turkish accuracy over the matched monolithic and self-consistency baselines? If not, the central idea is wrong. Second, does the full system beat single-model program repair at equal budget, especially on contrastive probes that differ in one conditioning property? If not, role separation is unnecessary here. Third, at matched data, does the system beat 52.6% on Egyptian Arabic and approach 77.4% on Lithuanian? That would decide whether this is a practical way to build inflection modules for low-resource languages or only a diagnostic for language model morphology.