Do Paradigm-Filling Models Over-Smooth? Contrast-Preserving Propagation for Inflection Tables

A verb in Finnish or Latin has an inflection table, or paradigm: dozens or hundreds of slots ("cells"), each holding the form for one combination of tense, person, number, mood and so on. Lexicographers working on low-resource languages usually have partial tables, with a few cells attested and the rest to be predicted. A natural way to fill them is to let the known cells pass information to the unknown ones along the table's structure, which is a graph whose edges link cells that differ in one grammatical feature. Graph neural networks, which update each node by averaging over its neighbors, have a well-studied failure here. Repeated averaging makes node representations converge, and the model loses the ability to tell nodes apart. This is called over-smoothing. Paradigms seem especially exposed to it, because neighboring cells share almost all their material (the stem) and differ only in a small, crucial affix. A model that smooths will output a neighbor's form, and a model that has collapsed onto the stem will copy whatever form dominates the table.

This note makes one testable claim and one proposal. The claim is that current paradigm-filling models fail in this way: as more known cells are supplied, accuracy plateaus or falls. The errors are overwhelmingly cases where the model produces another cell's correct form, and the internal representations of different target cells of the same word lose dimensionality. The proposal is a propagation model over the paradigm graph that carries two signals. One is a smoothing channel for the shared stem. The other is a "difference" channel for what separates a cell from its neighbors, mixed per cell. We predict its advantage over standard aggregation grows with a language's paradigm size and its syncretism rate, meaning how often distinct cells share one surface form. If the baselines show no such collapse, that negative result is itself informative and the proposal stops there.

What exists and what is missing

On the morphology side, three lines of work treat paradigm structure without any graph learning. Leveraging Principal Parts for Morphological Inflection fills cells with a character Transformer fed one or two source forms (1-src, 2-src) instead of only the dictionary form. It ranked first on SIGMORPHON 2020, but it never goes beyond two sources, so it cannot show whether adding more known cells helps or hurts. Frugal Paradigm Completion builds cell-to-cell predictability matrices and learns which cells to ask a human for. It shows predictability is strongly asymmetric, and that networks exploit stem-internal phonology instead of learning inflection classes. Principal Parts Detection for Computational Morphology: Task, Models and Benchmark clusters cells by edit scripts or reinflection accuracy and identifies which cells predict the rest. Its message passing is non-neural Affinity Propagation over precomputed matrices. These papers motivate the work: they establish that paradigms have rich relational structure. None of them propagates learned representations over that structure or measures whether representations collapse. Contextualization of Morphological Inflection supplies a design principle we reuse: keep the grammatical intent (which cell) separate from the surface realization (the characters).

On the graph side, the methods come from the over-smoothing literature. Complete the Missing Half adds a high-pass "diversification" channel next to ordinary neighbor averaging, with learned weights, and gains roughly 10 points on heterophilic graphs, where linked nodes tend to differ. Geometric Scattering Attention Networks chooses per node between low-pass and band-pass channels. When Do GNNs Work gates aggregation off when neighbors are unhelpful. PairNorm shows that propagation is necessary when many nodes lack input features, and that unnormalized deep propagation collapses in that regime. Partial paradigms are exactly this case: unknown cells have no form, only a tag.

Two papers constrain the design. Are we measuring oversmoothing in graph neural networks correctly? shows that the usual smoothness measure (Dirichlet energy) misleads in trained models, while effective rank tracks accuracy well (average Pearson correlation 0.91). It also warns that high rank is necessary but not sufficient. We therefore never report rank alone; we always pair it with an error typology. Alleviating Over-smoothing for Unsupervised Sentence Representation shows that Transformer layers also collapse, with cosine similarity above 0.9 between final layers. A fully connected Transformer over source cells is therefore not automatically immune, and it is the main thing we must beat.

Gnns have reached language before, in parsers such as Graph-based Dependency Parsing with Graph Neural Networks. Those graphs connect tokens in a sentence, use shallow networks and treat morphology as fixed input tags. The graph here is the paradigm itself, and collapse within that graph is the object of study.

Core Claim

Paradigm completion is a heterophilic graph problem with missing node features. Cells joined by an edge share a stem but must differ in exponence, meaning the affix that marks the grammatical feature. Aggregation that only averages, whether in a graph network or in attention over concatenated sources, moves target-cell representations toward the stem and the most frequent neighbor. This should appear as three things: neighbor-copy errors, a fall in effective rank across target cells of a lexeme, and diminishing or negative returns as the number of known cells $k$ grows. A difference channel defined by the table's own oppositions should remove most of this. The benefit should be largest where syncretism and paradigm size are largest.

Research Program

Building the graph. For each language we build a cell graph from UniMorph tags (UniMorph 4.0). Nodes are the cells of the paradigm schema, and an edge joins two cells whose feature bundles differ in exactly one feature. Where 4.0 provides hierarchical tags, as in case stacking or polypersonal agreement, the difference is computed over the tree, not a flattened string.

Initializing the nodes. A shared character encoder embeds each known cell's form together with its tag. Unknown cells start from the tag embedding only.

Propagation. Each of $L$ layers updates a cell $c$ as

$$h^{l+1}_c = \sigma\big(a_c\,[\hat A h^l W_L]_c + (1-a_c)\,[(I-\hat A) h^l W_H]_c\big) + h^l_c ,$$

where $\hat A$ is the normalized adjacency of the cell graph, $(I-\hat A)$ extracts the difference between a cell and the average of its neighbors, $W_L$ and $W_H$ are learned matrices, $\sigma$ is a nonlinearity, and $a_c\in[0,1]$ is a per-cell gate computed from $h^l_c$. The filterbank decomposition follows Complete the Missing Half, and the per-node gate follows Geometric Scattering Attention Networks.

Decoding. The target cell's final embedding conditions a character decoder that also attends over the known source strings. This keeps the grammatical intent separate from the characters, as in Contextualization of Morphological Inflection.

Instrumentation. Throughout training and across depths 1 to 8, we track the effective rank of the matrix of target-cell representations within each lexeme. We also track the neighbor-copy rate: the fraction of errors whose output equals the gold form of a different cell in the same table.

Baselines

The naive reference point is a copy heuristic. For each unknown cell, it outputs the known form of the nearest cell in the graph, breaking ties by frequency. It measures how much of the task is solved by syncretism alone. Next to it sits the lemma-only character Transformer that SIGMORPHON 2020 baselines represent.

The strongest existing method is multi-source paradigm cell filling from Leveraging Principal Parts for Morphological Inflection. That paper reports 91.72% average accuracy for 1-src on development data for 24 low-resource SIGMORPHON 2020 languages, and 91.63% for 2-src on 7 languages with fewer than 2,000 training examples. We extend it to $k$ sources by concatenating all known cells with their tags. This turns it into fully connected attention, which is the version a skeptic would actually propose. No reported result was found on Lacuna for paradigm filling with more than two sources, so we must establish this baseline's $k$-curve ourselves.

All systems use the same tables, the same lexeme-disjoint train/dev/test splits, the same sampling of which cells are known at each $k$, matched parameter counts and an equal hyperparameter search budget. That budget is 30 trials per system per language family, selected on development accuracy.

Two ablations isolate the key ingredient. The first removes the difference channel, setting $a_c=1$, which gives a plain residual graph network with PairNorm added as a separate variant. The second keeps both channels but replaces the one-feature-opposition graph with a complete graph. This tests whether the paradigm's structure matters or only the extra channel does.

Beating the strongest baseline means three things together. Exact-match accuracy must be higher at large $k$ across the language pool. Neighbor-copy errors must drop significantly. And the accuracy-versus-$k$ curve must keep rising where the baseline's flattens. A win only at $k\le2$ would not support the claim.

Why not just…?

Why not just give the Transformer all known cells and let attention sort it out? This is the real threat. Attention can learn to attend selectively, and it is not limited to averaging. Two things argue against assuming it works. The sentence-encoder results show attention stacks also collapse, and no paper in the file reports multi-source behavior beyond two sources. The $k$-sweep settles the question directly. If the concatenated Transformer's accuracy keeps rising with $k$, its neighbor-copy rate stays flat and its target-cell rank holds, then the premise is false for attention models. We would report the graph model only as an equal, cheaper alternative, or not at all.

Why not just condition the decoder more strongly on the target tag? Over-smoothing could be a weak-conditioning problem rather than a propagation problem. We test this with a baseline that feeds the target tag to every decoder layer and to the source encoder. The difference channel adds something tag conditioning lacks: the observed form difference along each known opposition, such as "how this lexeme's 2sg differs from its 3sg." A gain of tag conditioning that matches ours would show that the graph is unnecessary.

Why not just pick the single most predictive known cell and run 1-src? Frugal Paradigm Completion and the principal parts work show that a few cells predict most of a table. Choosing the best source per target from a predictability matrix might therefore beat any aggregation. We run this both with an oracle choice, the best source in hindsight, and with a learned choice. If the oracle single source matches our model, aggregation adds nothing. If our model exceeds the oracle on irregular or fragmented paradigms such as Latin, combining sources has real value.

Why not just use a standard shallow GNN with PairNorm? PairNorm keeps representations spread out but does not make them carry the right contrasts. On standard benchmarks it flattens degradation curves without raising peak accuracy. Our ablation with PairNorm and no difference channel shows whether rank preservation alone suffices.

Proposed Experiments

The data are UniMorph inflection tables for the 10 verbal paradigm languages of the principal parts benchmark (Hebrew, English, French, German, Spanish, Danish, Swedish, Finnish, Turkish, Latin). We add the 7 languages of Frugal Paradigm Completion, which include Arabic, Russian, Hungarian and Irish. We also include the SIGMORPHON 2020 languages, so that the 1-src and 2-src setting can be reproduced as a sanity check against the reported 91.72%. For each test lexeme we reveal $k\in\{1,2,4,8,\dots\}$ cells, sampled uniformly and also sampled to mimic corpus frequency, and predict the rest. Metrics are exact-match accuracy, Levenshtein distance, neighbor-copy rate, and within-lexeme effective rank of target representations.

Three checks decide the outcome. The first tests the premise: does the strongest baseline show a falling or flat curve as $k$ grows, together with rising neighbor-copy errors and falling rank? If not, we stop and publish the negative finding. The second tests the dose-response prediction. Across languages, our model's gain over the baseline should correlate positively with syncretism rate and paradigm size, both computed directly from UniMorph. English and Danish should show almost no gain, while Finnish, Turkish and Latin should show the most. The third tests the mechanism. Removing the difference channel, or replacing the opposition graph with a complete graph, should bring back the neighbor-copy errors.

As a secondary analysis, we read off which known-cell sets most raise target-cell rank. We score those cells against the gold principal parts of the detection benchmark, where the best reported method reaches 55.05% average F1 against a 21.20% random baseline. This step is exploratory and does not count toward the main claim.