Cutting Where Errors Don't Travel: Cascade-Guided Structure Regularization for Hierarchical Taggers

A structured predictor labels many interdependent parts at once, such as every word in a sentence or every nested entity in a span. These models can overfit by memorizing rare combinations of labels across long stretches of input, and shrinking their weights does not stop this. One known remedy, structure regularization, chops each training sentence into short random pieces so the model cannot condition on long, idiosyncratic label patterns. At test time the model still decodes the whole sentence. Hierarchical models, which predict in stages (inner entities first, then the entities that contain them), fail in a second way: a mistake at one stage becomes the input to the next and spreads. This note argues that the two failures are linked through one quantity, namely how far a local decision's influence travels through the structure. We propose to measure that influence during training and use it to decide where to cut.

The proposal is to replace random cuts with cuts placed where the model's own predictions are nearly independent across the boundary. At those points a cut removes long-range capacity while discarding almost no real dependency. At points where influence is strong, the "structural anchors" where errors cascade, a cut would teach the model a distorted picture of the dependencies. We test this on layered nested entity recognition and dependency parsing. In both settings cascading errors have been measured directly, so we can check whether the regularizer reduces cascades and not only overall error.

Core Claim

Structure Regularization for Structured Prediction supplies the method we build on. It proves that the generalization gap grows with the structural size of training samples. It decomposes each training sample into random mini-samples and leaves the choice of where to cut as an open heuristic. Identifying Cascading Errors using Constraints in Dependency Parsing supplies the motivation. It forces gold decisions on specific constructions at decode time and counts how many other arcs are repaired as a side effect. The costs turn out to be very unequal. Constraining punctuation raised accuracy by up to 1.8 points, and almost all of that gain (Δu = 1.5) came from fixing other arcs. Root errors displaced structure by more than nine words on average. Read together, these results say that random cuts treat every position as equally dependent, when some positions carry most of the influence.

Our claim is that a cut has two effects. It lowers capacity, which is the variance term Sun's bound captures. It also adds bias in proportion to the dependency it severs. The bias can be estimated from the model's own predictions. We define the cascade cost of cutting sentence $x$ at position $t$ as

$$c(t) = \sum_{i \in W_t} \sum_{\ell} \mathrm{KL}\big(p_\ell(y_i \mid x)\ \|\ p_\ell(y_i \mid x_{\text{seg}(i,t)})\big),$$

where $W_t$ is a window of tokens on either side of $t$, $\ell$ indexes the stages of the hierarchy, and $p_\ell(y_i \mid x)$ is the model's label distribution for token $i$ at stage $\ell$ given the full sentence. The term $p_\ell(y_i \mid x_{\text{seg}(i,t)})$ is the same distribution when the model sees only the segment containing $i$ after the cut. We then sample cut positions with probability proportional to $\exp(-c(t)/\tau)$. Taking $\tau \to \infty$ recovers Sun's random decomposition, positive $\tau$ prefers low-cascade cuts, and negative $\tau$ prefers anchor cuts.

The sum over stages $\ell$ is what ties the method to hierarchical error propagation. In a layered tagger, a cut that barely changes inner-layer predictions can still change outer-layer ones, because outer layers pool over inner spans. The cost therefore counts influence that reaches higher levels.

Research Program

The main testbed is the layered tagger from A Neural Layered Model for Nested Named Entity Recognition. It stacks flat sequence taggers, each pooling the hidden states of entities found by the layer below. It is the clearest example of bottom-up cascading: if the innermost entity is missed, the enclosing one cannot be found. The original model uses only dropout and weight decay, with no structural control. We rebuild it with a pretrained transformer encoder in place of word embeddings, so the results speak to current representations.

During training we recompute $c(t)$ every few epochs on each training sentence. This needs one full forward pass plus one pass per candidate segment, run on GPU in batches. To keep the cost bounded, candidates are restricted to positions outside gold entity spans. Within those positions we compare low-cascade, high-cascade and uniform sampling, with the length distribution of the resulting mini-samples held identical across conditions. Holding length fixed is essential: without it, any gain could simply come from shorter samples rather than better-placed cuts.

The second testbed is the global span model of Nested Named Entity Recognition with Partially-Observed TreeCRFs. It treats nested entities as a partially observed constituency tree, scores spans with a transformer, and already uses its own smoothing and normalization regularizers. It has no stage-to-stage cascade, so it is a useful control: a gain there would come from reduced overfitting alone. Decomposition also cuts its cubic-time training cost from $n^3$ to a sum of smaller cubes.

The third testbed is dependency parsing. Here Ng and Curran's constrained-decoding protocol becomes an outcome measure and not only a diagnostic. If cascade-guided training works as claimed, the Δu attributable to punctuation and noun-phrase attachment should fall, meaning the trained parser's errors spread less. As a sanity check, we also run the flat tasks Sun used (POS tagging and BioNLP-2004 entity recognition) to confirm that the guided sampler does no harm where hierarchy is absent.

Baselines

The naive reference point is the layered tagger with dropout and weight decay tuned on the development set, as in Ju et al. That paper reports 74.7 F1 on GENIA and 72.2 F1 on ACE2005 with a recurrent encoder. Because we change the encoder, we rerun this baseline ourselves and treat those figures only as a historical anchor.

The most direct competitor is Sun's random decomposition, reimplemented on the same encoder. Its mini-sample size is tuned with exactly the same search budget as our temperature $\tau$ and window $W_t$. Under these conditions Sun reports 97.36% POS accuracy for structure regularization; we found no reported result on Lacuna for structure regularization on nested NER.

The strongest existing system on these benchmarks is the PO-TreeCRF, which reports 85.4 F1 on ACE2005 and 86.6 F1 on ACE2004 (Fu et al.). We do not expect a layered tagger to beat it outright. Instead we apply our sampler to the TreeCRF itself, using the same encoder, data splits and number of seeds. Beating it means a gain over its reported figures and over our own reproduction that exceeds the spread across five seeds.

Two ablations isolate the key ingredient. Matched-length uniform cuts test whether cut placement matters at all. Negative-$\tau$ anchor cuts test the direction of the effect. A further heuristic baseline cuts only at punctuation, which uses Ng and Curran's finding without any model-based measurement.

For the layered model, beating the random-decomposition baseline means three things together:

  • higher outer-entity F1 at nesting depths three and above, where Ju et al. report sparse data and fast degradation;
  • a lower conditional cascade rate, defined as the fraction of outer entities missed given that their inner entity was missed;
  • no loss on flat entities.

Why not just…?

Why not just tune weight decay and dropout harder? Sun's analysis shows that norm penalties leave the structural term of the bound untouched, and his decomposed CRFs beat L2-tuned ones. That evidence comes from linear models, though, and transformer encoders may behave differently. The naive baseline receives the same tuning budget as our method, so a null result here would be informative.

Why not just train upper layers on predicted rather than gold lower layers, as in stacking? Stacked Hierarchical Labeling trains each child classifier on held-out parent predictions so that it learns to discount wrong context. This is a serious threat, because it targets the train/test mismatch that drives cascades directly. Our view is that stacking fixes how upper layers react to errors, while decomposition reduces how confidently lower layers make them, so the two should add. We run out-of-fold stacking for the layered tagger, then stacking combined with our sampler. If stacking alone matches the combination on the conditional cascade rate, the cascade half of our claim fails, and the proposal reduces to an overfitting regularizer.

Why not just use a global model that has no cascade? The TreeCRF removes stage-to-stage propagation but still overfits rare long spans; its authors add potential normalization for that reason. Applying our sampler to it tests whether the overfitting benefit holds without the cascade benefit.

Why not just cut at punctuation or other obvious boundaries? This is cheap and may capture most of the effect. The punctuation-only baseline is included precisely so that the experiment can show whether measuring $c(t)$ adds anything beyond the linguistic prior.

Decisive Checks

We use GENIA, ACE2004 and ACE2005 for nested entities, an English dependency treebank for the parsing experiments, and Sun's POS and BioNLP-2004 tasks as flat controls. No Lacuna resource entries exist for these corpora, so they will be obtained from their standard distributions.

The central check is the three-way comparison of low-cascade, uniform and high-cascade cuts at matched mini-sample lengths. The proposal predicts that results improve in that order. If uniform and low-cascade cuts are indistinguishable, the claim fails.

A second check measures the trained parsers with Ng and Curran's protocol. Lower punctuation and noun-phrase Δu under guided training would show that the regularizer changed how far errors spread, not just how many there are.

A third check plots conditional cascade rate against nesting depth for the layered tagger under each training scheme, with stacking included. This shows directly whether reducing lower-layer overfitting reduces upward propagation.

Finally, we report the training cost of computing $c(t)$ against the speedup that decomposition gives. If the bookkeeping costs more than it saves, that limits where the method is worth using.