Many language-analysis systems run two kinds of model on the same sentence. The first is a word-level tagger that labels each word's part of speech and grammatical features. The second is a character-level generator that writes out each word's dictionary form (its lemma) or its fully vowelled spelling (its diacritized form). Training both jointly on shared representations tends to fail, because the generator's loss is noisy and pulls the tagger's features around. The best-known fix, from Joint Diacritization, Lemmatization, Normalization, and Fine-Grained Morphological Tagging, inserts a stop-gradient: a one-way valve that lets the generators read the tagger's predictions but blocks their training signal from flowing back into it. That fix has two costs. The tagger never learns anything from the lexical knowledge the generators hold, and nothing prevents the two halves from contradicting each other, for example by pairing a noun tag with a verb lemma.
We propose replacing the all-or-nothing valve with a narrow, structured channel. The tagger's output layer becomes a linear-chain conditional random field (CRF), a model that scores whole label sequences rather than one word at a time. We add one extra term to it: a compatibility score between each word's tag and a small probability vector that the lemma generator emits over coarse lexical classes. The generator's full character-level loss stays blocked from the tagger. Only the CRF's globally normalized, low-dimensional gradient crosses between the two models, and the joint decoding step rules out tag–lemma contradictions. Our hypothesis is that negative transfer comes from the variance and dimensionality of the generator's gradient, not from the information it carries. If that is right, a bottlenecked channel shaped by the CRF should recover positive transfer that the plain stop-gradient throws away, without bringing back the instability.
Core Claim
A CRF-mediated gradient channel between a word tagger and a character generator will do three things at once. It will cut tag–lemma contradictions to near zero. It will improve tagger and lemmatizer accuracy over the static stop-gradient of Zalmout and Habash. And it will do so because of the gradient that flows through the channel, not merely because the two heads are coupled in the forward pass. The last part is the contribution that is actually new, and one ablation below is designed specifically to test it.
Mechanism
The tagger's encoder produces a state $h_t$ for word $t$. The lemma decoder is a character-level generator conditioned, as in Zalmout and Habash, on stop-gradded tagger features. At the end of each word it emits a distribution $q_t$ over $K'$ coarse lexical classes (open-class verb, noun, adjective, particle, and so on) from a small projection of its final state. We score a sequence of part-of-speech tags $Y=(y_1,\dots,y_T)$ as
$$s(X,Y)=\sum_{t=1}^{T}\Big[\psi(y_t,h_t)+A_{y_{t-1},y_t}+C_{y_t}^{\top}q_t\Big],$$
The three terms have simple roles. $\psi$ is the tagger's per-word emission score. $A$ is the learned tag-to-tag transition matrix. $C$ is a learned tag-by-class compatibility matrix, whose row $C_{y_t}$ scores how well tag $y_t$ fits the decoder's class distribution $q_t$. Training minimizes the usual CRF negative log-likelihood of the gold tag path, plus the decoder's character loss.
The routing of gradients is the point of the design. The character loss reaches the decoder only. The CRF loss reaches the tagger through $\psi$. It also reaches the decoder, but only through the $K'$-dimensional vector $q_t$. The gradient with respect to $q_t$ is $C$ times the difference between the model's expected tag and the gold tag, so it is bounded and has already been averaged over all tag sequences. At inference, Viterbi decoding picks the tag path jointly with the decoder's class evidence. We then rerank the decoder's beam of lemmas under the chosen tag, so the reported pair is consistent by construction.
We run the CRF over part-of-speech tags only, roughly tens of classes. Full Arabic morphological tag tuples number in the thousands and would make the transition matrix unwieldy. Finer features (gender, number, case) stay as local heads, as in the original system.
Relation to Prior Work
Joint Diacritization, Lemmatization, Normalization, and Fine-Grained Morphological Tagging supplies the setting and the baseline. It builds a multi-task tagger with lemma, diacritization and orthographic-normalization decoders, and a stop-gradient from tagger to decoders. It reports a 20% relative error reduction on Modern Standard Arabic and 11% on Egyptian Arabic over earlier LSTM systems. Its tagger has no CRF, its isolation is total and static, and it produces contradictory outputs. We keep its isolation of the character loss but open a controlled path in both directions.
Two other papers motivate the CRF choice. Structured prediction models for RNN based sequence labeling in clinical text shows that per-word classifiers fragment labels while CRF variants fix this. Its approximate skip-chain CRF reaches 0.8210 strict F. It also shows that context-dependent pairwise potentials help precision, which is a precedent for adding learned potentials beyond a fixed transition matrix; our compatibility term is a unary cross-model potential rather than a pairwise one. Bregman Conditional Random Fields: Sequence Labeling with Parallelizable Inference Algorithms evaluates CRFs only as single-task heads. Its parallel inference (Lightspeed CRF) is an optional drop-in if sequential Viterbi becomes a bottleneck. The GPL-3.0 license matters if we release code.
Two further papers are relevant in narrower ways. Multi-task Learning with CTC and Segmental CRF for Speech Recognition combines a CRF with another sequence objective, but with plain joint gradients. It therefore sits at the "no isolation" end of our spectrum. Regression over Classification: Assessing Image Aesthetics via Multimodal Large Language Models and Unifying Deep Local and Global Features for Image Search use the same binary stop-gradient idea in vision. We treat them as evidence that the binary valve is the default design, not as methods we reuse.
Research Program
We first reimplement the Zalmout and Habash architecture on the Penn Arabic Treebank (Modern Standard Arabic) and ARZ (Egyptian Arabic). We verify that joint training without isolation degrades tagging, as they report. We then swap the tagger's part-of-speech head for a linear-chain CRF with no cross-model term, which isolates the value of the CRF alone. Next, we add the compatibility channel and train under the gradient routing described above.
Alongside accuracy, we log two gradient diagnostics on the shared tagger encoder, reported by layer across training. The first is the variance of the gradient each loss source contributes. The second is the cosine between that contribution and the gradient of the tagger's own loss. These diagnostics test the mechanism directly: the channel should show low variance and a non-negative cosine, and an unisolated character loss should not.
To check that the result is not specific to Arabic, we repeat the core comparison on Universal Dependencies 2.15 languages that provide lemmas and part-of-speech tags. We use the English, French, German and Dutch treebanks that the Bregman CRF paper already evaluates for tagging.
Baselines
The naive reference point is plain joint training. Tagger, CRF and decoders share gradients with no stop-gradient. This is the setup Zalmout and Habash found unstable, and it tells us how much any isolation is worth.
The strongest existing method is the Zalmout and Habash static stop-gradient system, reimplemented with the same encoder, data splits, and tuning budget as ours. On this benchmark the file gives only relative figures for it: 20% relative error reduction on Modern Standard Arabic and 11% on Egyptian Arabic versus earlier LSTM models, plus outperforming MADAMIRA. No absolute accuracy for it was found on Lacuna. We therefore compare against our own reproduction, and we only report our numbers if the reproduction matches the paper's reported gains over its LSTM baseline. Beating it means a statistically significant gain in joint all-features-correct accuracy and in lemma accuracy across at least three seeds, with no loss in part-of-speech accuracy, and a contradiction rate near zero.
A third baseline is a CRF-only variant: the static stop-gradient system with a CRF tagger head but no compatibility term. It separates the benefit of structured tagging from the benefit of the channel.
The key ablation keeps the compatibility term in the forward pass and in joint decoding but places a stop-gradient on $q_t$ and on the tagger side of $C$. The two models are then coupled at decode time but receive no training signal from each other. If this ablation matches the full model, the gain is joint decoding, not gradient routing, and our core claim about gradients is false.
A second ablation widens the channel from $K'$ classes to the decoder's full hidden state. This tests whether the bottleneck is what keeps training stable.
All baselines receive identical hyperparameter search budgets and the same number of training steps.
Why not just…?
Why not just scale the generator's gradient by a small factor instead of stopping it? This is the most serious threat. A partial-gradient multiplier on the path from decoder to tagger might recover positive transfer as well. We sweep the multiplier over several orders of magnitude, with the same tuning budget as our method. If some value matches our method on accuracy, the gradient-channel claim reduces to a tuning trick. The multiplier alone still cannot remove contradictions; that advantage would then belong to the joint decoding step. The variance diagnostics will show whether a scaled high-dimensional gradient behaves like our bottlenecked one.
Why not just release the stop-gradient after a warmup period? Scheduled isolation addresses the "frozen tagger" problem with no new parts. We run it with release points at 25%, 50% and 75% of training. The reason it may fall short is that the decoder's loss stays high-variance late in training. If scheduled release matches our method, we will report that as a finding.
Why not just filter contradictory tag–lemma pairs at inference with a lexicon? A post-hoc reranker that checks tag–lemma pairs against lemma–POS pairs from a morphological analyzer or the training data would likely remove most contradictions for free. We include it applied to the static stop-gradient system. It cannot improve the tagger's learned representations, so any accuracy gap between it and our model on words the lexicon does not cover is evidence for the training-time channel. If there is no such gap, the consistency half of our contribution is not needed.
Why not just replace the tagger with a sequence-to-sequence model? Exploring Sequence-to-Sequence Learning in Aspect Term Extraction shows that autoregressive decoders learn which label transitions are legal without a transition matrix. That does not solve the problem here. Folding tags and lemmas into one generated stream exposes the tagger to exactly the character-level loss variance that forced Zalmout and Habash to isolate it, and it decodes one token at a time. We treat this as out of scope rather than as a competitor on this specific question.
Decisive Checks
We will measure part-of-speech accuracy, full morphological tag accuracy, lemma accuracy, diacritization accuracy, and joint all-correct accuracy on the Penn Arabic Treebank and ARZ. We will also measure the tag–lemma contradiction rate, defined against analyzer lemma–POS pairs, plus the gradient variance and cosine diagnostics.
Three results would settle the proposal. The first is whether the full model beats the forward-only coupling ablation: this is the test of the gradient claim. The second is whether it beats the best gradient-scaling multiplier: this is the test of the bottleneck claim. The third is whether it beats lexicon filtering on words the lexicon does not cover: this is the test of whether representations improved. The Universal Dependencies replication checks whether any gain holds outside Arabic, where morphology is less rich and the compatibility term may matter less. We expect smaller gains there and will report them either way.
References
- Joint Diacritization, Lemmatization, Normalization, and Fine-Grained Morphological Tagging
- Structured prediction models for RNN based sequence labeling in clinical text
- Bregman Conditional Random Fields: Sequence Labeling with Parallelizable Inference Algorithms
- Multitask Learning with CTC and Segmental CRF for Speech Recognition
- Regression over Classification: Assessing Image Aesthetics via Multimodal Large Language Models
- Unifying Deep Local and Global Features for Image Search
- Exploring Sequence-to-Sequence Learning in Aspect Term Extraction