Keeping Only the Tag Splits a Parser Cannot Already See

Part-of-speech tags such as "noun" or "determiner" are too coarse for parsing. Müller et al. showed that an unsupervised procedure can refine them, splitting each tag into finer subtags (for example, German articles by grammatical case) and feeding those subtags to a dependency parser. The procedure decides which splits to keep by asking only whether a split makes the tag sequence more probable under a simple sequence model. The parser itself, which already sees the actual words, never gets a say. The authors note the consequence themselves: some splits capture local patterns the parser already handles. We propose changing the keep-or-discard decision. A split survives only if it tells us something about the syntactic attachment of the word that the word and its neighbours do not already tell us.

The claim is that the right score for an iterative tag-refinement procedure is conditional information about syntax given lexical context, not likelihood gain. If that is right, the method should reach the parsing gains of likelihood-driven refinement with far fewer subtags. It should also keep a measurable gain in the regime where likelihood-driven refinements are expected to stop helping, namely parsers built on contextual encoders. Both predictions can fail cleanly, and either failure would itself be informative about where residual syntactic signal lives once a model has access to words.

Background and Positioning

Two observations in the literature motivate the proposal. An Information-Theoretic Empirical Analysis of Dependency-Based Feature Types for Word Prediction Models measures how much information each feature type carries for predicting the next word. The previous word alone carries 3.826 bits, and adding the word it modifies adds 0.683 bits, yet POS tags add only 0.002 bits once word identities are known. Coarse syntactic categories are almost entirely redundant with lexical identity. Recursive Subtree Composition in LSTM-Based Dependency Parsing finds that explicit tree composition adds negligible accuracy once a bidirectional LSTM reads the sentence. Hand-built structure helped only when the forward LSTM or the POS tags were removed. Together these say that any added syntactic feature should be judged by what it adds beyond the lexical representation already present.

The method comes from Dependency parsing with latent refinements of part-of-speech tags, which builds a hidden Markov model whose hidden states are refined subtags. Its split-merge training uses Expectation-Maximization (EM), an iterative procedure that alternates between inferring the hidden subtags and re-estimating model probabilities. It splits every tag in two, re-estimates, and merges back splits whose likelihood gain is smallest. On CoNLL-2009 this raised English labeled attachment score (LAS, the share of words assigned the correct head and relation label) from 90.34 to 90.57, and German from 87.92 to 88.24. The German result is comparable to the 88.35 obtained by adding hand-engineered morphology. We keep its EM machinery intact and change only the merge criterion. The proposal therefore differs from Müller et al. in exactly one place, and that difference is what we test.

Two other EM papers constrain the design. Lateen EM shows that marginal likelihood correlates poorly with syntactic accuracy in grammar induction, which is the general form of the misalignment we target. Unified Expectation Maximization shows that softer and harder posteriors trade off differently across tasks. We therefore treat the sharpness of subtag posteriors as a tuned knob rather than fixing hard assignments. Neither paper changes what EM is asked to keep; that is our contribution.

Core Claim

Let $Z$ be a candidate subtag assignment for a token. Let $Y$ be the token's attachment attributes in the training treebank: head direction, distance bucket, relation label and the head's coarse tag. Let $C$ be the lexical context the downstream parser already sees, meaning the word form and its immediate neighbours. We score a split of tag $t$ by the held-out conditional information gain

$$\mathrm{IG}_t = \mathbb{E}\left[\log q(Y \mid C, Z) - \log q(Y \mid C)\right],$$

where $q$ is a fixed, regularized log-linear predictor of $Y$, and the expectation is over held-out tokens of tag $t$ with $Z$ drawn from the EM posteriors. This is the empirical counterpart of the conditional mutual information $I(Z;Y \mid C)$, in the spirit of Wu et al.'s information-gain measures. Splits with the lowest $\mathrm{IG}_t$ are merged.

Because $C$ already contains the words, a split that merely rediscovers lexical classes scores near zero even when it improves likelihood. When the downstream parser has a contextual encoder, $C$ becomes that encoder's frozen token representation. The criterion then asks what the subtag adds beyond the encoder.

Research Program

We build split-merge refinement exactly as in Müller et al.: the same number of split-merge rounds, the same smoothing, and the same parser integration with subtags conjoined into arc features. We then substitute three merge criteria. The first is likelihood gain, which reproduces the original method. The second is unconditional task information $I(Z;Y)$. The third is our conditional criterion $I(Z;Y \mid C)$.

The second criterion is the ablation that isolates our key ingredient. Giving EM any syntactic feedback is an obvious idea. The question is whether it is the conditioning on lexical context, the redundancy correction, that matters. A fourth variant goes further than changing the merge rule. It lets the hidden Markov model also emit $Y$ during training, so the E-step itself is syntax-informed. We check whether pushing the signal into estimation helps beyond pushing it into selection. The posterior sharpness parameter from Unified EM is tuned on development data for every variant under an equal budget.

We run two parser regimes. One is the lexicalized feature-based graph parser used by Müller et al., where their numbers give a direct reference. The other is a bidirectional-LSTM graph parser, where de Lhoneux et al.'s findings predict that likelihood-driven subtags give roughly nothing. The analysis side reports $\mathrm{IG}_t$ per tag and inspects surviving splits for linguistic content: case on German articles, and infinitival versus prepositional "to" in English. It also measures how much of each split's information the BiLSTM states already carry, by re-estimating $\mathrm{IG}_t$ with $C$ set to those states.

Baselines

The naive reference point is the same parser with coarse gold-scheme tags and no refinement. On CoNLL-2009 Müller et al. report 90.34 LAS for English and 87.92 for German without morphology. Every method shares the same training sections, parser and feature templates, so the reference costs nothing extra to reproduce.

The strongest existing method is likelihood-driven split-merge refinement from Dependency parsing with latent refinements of part-of-speech tags. It reports 90.57 LAS for English and 88.24 for German without morphology, with a best tag inventory of about 115 for English and 107 for German. We rerun it under our pipeline so that differences come only from the merge rule. Each criterion gets the same number of EM iterations and rounds and the same development-set tuning grid over inventory size and posterior sharpness. Beating it means one of two outcomes. The first is matching its LAS (within the seed-to-seed spread across five runs) with substantially fewer subtags, which we expect around half. The second is exceeding its LAS at the same inventory size by more than seed variance.

The ablation baselines are unconditional task information and the syntax-emitting hidden Markov model described above. If unconditional task information matches our criterion, the redundancy story is wrong and the gain comes from task feedback alone.

Explicit morphology is the hand-built alternative. For German the file reports 88.35 LAS with morphology; we also combine it with each refinement variant, where Müller et al. report 88.51.

A greedy wrapper is the expensive upper reference: retrain the parser for each merge candidate and keep the split that helps development LAS most. It is only affordable on a subset of tags and a reduced training set. We report how closely our cheap criterion's rankings agree with the wrapper's.

For the BiLSTM regime, no reported result was found on Lacuna for CoNLL-2009 LAS with a BiLSTM parser using refined tags. There we compare only against our own reruns of the coarse-tag and likelihood-refinement baselines.

Why not just…?

Why not just drop tags and rely on a contextual encoder? This is the most serious threat. Recursive Subtree Composition in LSTM-Based Dependency Parsing is exactly the kind of evidence that says explicit structure stops mattering. The BiLSTM regime tests it directly. If neither likelihood nor conditional refinement improves the BiLSTM parser beyond seed variance, we report that syntactic tag refinement is fully subsumed in this setting. The project then becomes a measurement paper, quantifying through $\mathrm{IG}_t$ with encoder states as $C$ how little residual information tags carry. We do not try to argue around a null.

Why not just let the parser's regularizer prune useless subtag features? A sparsity penalty on parser weights can zero out redundant conjunctions, but only after the tag inventory is fixed. EM still spends its splitting budget on lexical distinctions, and the useful splits it never made cannot be recovered. We run likelihood refinement with an L1-regularized parser as an extra baseline. If it matches our method at equal inventory size, this objection wins.

Why not just choose splits by retraining the parser and checking development accuracy? That wrapper is the most direct task-aware selection rule, and on small problems it could be as good or better. Its cost grows with the number of candidate splits times parser training time. Our criterion is meant to be a cheap approximation of it. We report rank agreement and LAS against the wrapper on the subset where the wrapper is affordable. A large gap in the wrapper's favour would mean our proxy $q$ is too weak.

Why not just cluster word embeddings within each tag? Within-tag clustering of distributional vectors, Brown-style, is cheap and needs no EM. However, it optimizes lexical similarity, which is close to the very redundancy we are trying to remove. We include it as a baseline at matched inventory sizes and predict it behaves like likelihood refinement, scoring low on $\mathrm{IG}_t$ despite plausible clusters.

Decisive Checks

The experiments use CoNLL-2009 English and German, following Müller et al.'s splits, plus three or four typologically varied UD v2.1 languages from de Lhoneux et al.'s set for the BiLSTM regime. The main result is the LAS-versus-inventory-size curve for each merge criterion, averaged over five seeds.

Three findings would decide the question. First, the conditional criterion should dominate the likelihood criterion's curve on the feature-based parser. Second, it should beat the unconditional-information ablation; this isolates the redundancy correction. Third, in the BiLSTM regime, the conditional criterion's held-out $\mathrm{IG}_t$, computed with encoder states, should predict which languages and tags still yield LAS gains. The third check makes the work useful even if the gains are small. It turns Wu et al.'s descriptive redundancy measure into a predictive test of when explicit syntactic refinement is worth estimating at all.