Spelling-Anchored Decay: Regularizing Word Embeddings Toward Their Characters

Models that read text at two levels at once, whole words and their individual characters, almost always regularize both parts the same way. They apply one fixed weight decay, a penalty that pulls every parameter toward zero. That choice ignores how differently the two parts learn. A rare word's embedding row receives a handful of updates over all of training. A character embedding is updated on nearly every token. Decaying the rare word's vector toward zero throws away information the model already has, because the word's spelling, read by the character half of the model, says a good deal about what the vector should be.

We propose to change both what the penalty pulls toward and how hard it pulls. Each word embedding is pulled toward the vector that the model's own character encoder computes from the word's spelling, not toward zero. The pull on each row weakens as that row accumulates training updates. Rare words therefore stay close to what their spelling predicts. Frequent words are gradually freed to specialize. The penalty changes per row and over time, so a small grid of fixed coefficients cannot reproduce it. Whether that extra flexibility matters is the main empirical question, and the design below is built so it can come out negative.

Core Claim

Consider a hybrid model with word embeddings $e_w$ and a character encoder $g$ that maps the spelling $c(w)$ of word $w$ to a vector of the same size. We replace weight decay on the word table with

$$\mathcal{R}(t) = \sum_{w} \lambda_w(t)\,\big\|e_w - g(c(w))\big\|^2, \qquad \lambda_w(t) = \frac{\lambda_0}{1 + n_w(t)/\kappa},$$

where $n_w(t)$ is the number of training updates row $w$ has received by step $t$, $\lambda_0$ is the initial pull strength, and $\kappa$ is the update count at which the pull halves.

The $1/(1+n/\kappa)$ form comes from Gaussian shrinkage. Under a prior centered on the spelling-predicted vector, the weight that prior keeps in the posterior falls off as prior precision divided by prior precision plus data count. We test two variants: gradients from the penalty flow into $g$, or $g$ receives a stop-gradient. The first teaches the character encoder to predict word vectors. The second keeps the regularizer passive.

The claim we will test is narrow. Relative to the best tuned static regularization, including separately tuned decay for the word and character parameter groups, this penalty should lower perplexity on rare and morphologically complex words. It should also make models more robust to misspellings, with the gains largest on morphology-rich text and at small data sizes.

Relation to Prior Work

Hybrid character-word models motivate the setting.

  • Character-Word LSTM Language Models concatenates character embeddings with word vectors in an LSTM language model. It reports relative perplexity gains of up to 2.77% on Penn Treebank and 4.57% on Dutch CGN, using uniform optimization for both parts. We keep its architecture and change only the regularizer.
  • Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models makes a hard, frequency-thresholded switch, sending rare and unknown words to a character LSTM. Our penalty is a soft, continuous version of the same intuition: trust spelling when word-level evidence is thin. It lets each word move along that spectrum during training instead of being assigned to one side.
  • BennettNLP at SemEval-2021 Task 5 statically stacks character-level Flair features with BERT and GloVe for toxic span tagging, again with fixed hyperparameters.

Dynamic penalty methods supply techniques and constraints.

  • Amos already argues that sparsely updated parameters should be decayed less. It reports reaching AdamW's target loss in at most 70% of the steps on BERT and T5. But Amos decays toward zero and treats the vocabulary as one table. We share its sparsity argument and change the decay target.
  • Adaptive Weight Decay for Deep Neural Networks scales decay per parameter from layer-normalized gradient magnitudes. This assumes parameters within a layer have comparable gradient scales, which is false for a mixed word-character input layer. It was tested only on images. - On the Overlooked Pitfalls of Weight Decay ties decay to an average of squared gradients. It reports that this failed to beat plain L2 on a word-level LSTM on Penn Treebank. That result constrains us: global gradient-norm feedback is a poor signal in language models. So we key the schedule to per-row update counts, which are cheap and exact, not to gradient norms.

  • Task-adaptive Pre-training of Language Models with Word Embedding Regularization regularizes embeddings toward a reference with a fixed coefficient. It is the closest in form to our penalty, but it has no character target and no per-row schedule.

Research Program

We first build the penalty into the character-word LSTM language model. This architecture has a published comparison point and is cheap enough for full tuning sweeps. The character encoder $g$ is the model's existing character pathway, pooled to word dimension. Update counts $n_w(t)$ are kept as an integer vector, so memory overhead is one counter per vocabulary entry.

We then run a diagnostic study that also addresses the gradient-norm finding of Xie et al. We will log per-row gradient norms and parameter norms, binned by word frequency. This tests whether the late-training gradient spikes in word-level language models come mainly from rare rows. If so, that would explain why a global gradient-norm schedule misfires there and a per-row count schedule does not.

Finally, we transfer the penalty to a noisy-text tagging task, toxic span detection, to check that the effect is not confined to perplexity.

Baselines

The naive reference is the character-word LSTM trained with a single static weight decay coefficient on all parameters, grid-tuned. This is how Character-Word LSTM Language Models and nearly all hybrid models are trained. We also train a word-only LSTM so that the hybrid gain itself can be reproduced. The file reports only relative improvements for that paper (2.77% on PTB, 4.57% on CGN), not absolute perplexities. Our first check is to reproduce gains of roughly that size before measuring anything else.

The strongest competitor is two-group static decay. Here the word table and the character parameters each get their own fixed coefficient, tuned on a two-dimensional grid. This is what a careful practitioner would do. It already captures much of the "different statistics, different penalty" argument. It is the baseline the proposal most needs to beat.

We also run the model under Amos and under AdamS from On the Overlooked Pitfalls of Weight Decay. These are the strongest existing dynamic-decay optimizers, and they are run with their recommended settings plus a small tuning sweep. For neither do we have an absolute perplexity: no reported result found on Lacuna for Amos or AdamS perplexity on PTB. Xie et al. say only that their method lost to plain L2 there.

For toxic spans, the reference is the stacked-embedding system of BennettNLP, which reports 0.748 F1. The task paper SemEval-2021 Task 5: Toxic Spans Detection reports a winning character-based F1 of 70.83%. We will retrain the stacked model ourselves under each regularizer, not compare across papers with different scoring.

Three ablations isolate the ingredients:

  • Uniform pull toward the character vector with constant $\lambda$, which isolates the target.
  • Pull toward zero with the count-based schedule, which is lazy, frequency-aware weight decay and isolates the schedule.
  • A static frequency-scaled pull using final corpus counts in place of the running count $n_w(t)$, which isolates the time dynamics.

Fairness is enforced by giving every method the same data, architecture, and number of training steps. Every method also gets an identical budget of tuning trials, 24 configurations each, so our method gains nothing from having two hyperparameters where two-group decay also has two. Each configuration is run over three seeds.

Beating the strongest competitor means improving rare-word perplexity beyond the seed variance of the two-group baseline on both PTB and CGN, without hurting overall perplexity.

Why not just…?

Why not just tune separate static weight decays for word and character parameters? This is the real threat, and the experiments are designed around it. A two-group grid can learn that word rows deserve less decay than character weights. But it applies one coefficient to every word row. It cannot tell a word seen ten times from one seen ten thousand times, and it still pulls rare rows toward zero, not toward anything informative. If two-group decay matches our method on the rare-frequency bins, the per-row spelling-anchored penalty adds nothing and we will report that. The prediction that would separate them is a gap that grows as word frequency falls and as training data shrinks.

Why not just use Amos or AdamS, which already adapt decay to sparsity or gradients? Amos does encode the idea that sparse parameters should be decayed less. It is the closest existing mechanism, and the "pull toward zero with count schedule" ablation reproduces its sparsity logic within our setup. AdamS is directly undermined by its own authors' report that it lost to static L2 on a Penn Treebank LSTM. Running both optimizers as baselines settles whether spelling-anchoring adds anything beyond sparsity-aware decay.

Why not just hard-switch rare words to characters, as Luong and Manning do? Hard switching works for unknown words, and it reports +2.1 BLEU over identity-copying word models at a 50,000-word vocabulary. But it forces a binary choice at a fixed frequency threshold. Words just above the threshold get a poorly trained word vector and no character help. We include a thresholded-switch variant of the character-word LSTM. If it equals our method across frequency bins, the soft schedule is unnecessary.

Why not just use subword tokenization? Subwords shift the problem rather than remove it: rare subword rows are still sparsely updated and still decayed toward zero. Chinese Spelling Correction: A Comprehensive Survey documents that subword-based large models remain weak on character-level errors. We treat subwords as out of scope for the first study. The penalty transfers directly to them, with a subword row anchored to its characters.

Proposed Experiments

The primary experiments are language modeling on Penn Treebank and on CGN, whose richer inflection should favor a spelling anchor. We report overall perplexity and perplexity binned by training frequency. We also report perplexity on test sentences with synthetic character-level typos injected at fixed rates, which measures robustness to spelling noise. A data-size sweep trains on 10%, 30%, and 100% of each corpus. The Bayesian reading predicts the advantage over two-group decay shrinks as counts grow. If the advantage instead stays constant or appears only at full data, the shrinkage explanation is wrong.

The secondary experiment retrains the BennettNLP-style stacked tagger on the SemEval toxic spans split (5,339 training and 558 test examples) under each regularizer. Social-media misspellings are where character anchoring should help.

The gradient diagnostic is reported alongside the language models. It asks whether rare rows dominate late-training gradient spikes in the word-only model, and whether the anchored penalty removes those spikes.

The decisive check is a single comparison: our method against tuned two-group static decay on rare-bin perplexity on CGN at 10% data. If they are within seed variance, the core claim fails.

A second check comes from the time-dynamics ablation. If the running-count schedule does no better than final-count scaling, then the "dynamic" part of the penalty is unnecessary, and the contribution reduces to a static, frequency-scaled spelling prior.