Finding Label Errors in Sentence Compression Benchmarks One Subtree at a Time

Sentence compression means shortening a sentence by deleting words while keeping it grammatical and faithful. The field trains and scores systems against a single human "gold" compression per sentence, and treats that reference as correct. It often is not. When Gupta et al. (2009) checked human compressions against simple grammar rules, every rule was broken at least 10% of the time. Humans dropped the main verb in 34% of Ziff-Davis sentences. In 72% of the Clarke–Lapata broadcast-news pairs, a verb was deleted while its arguments were kept. Work on vision and text benchmarks shows what this kind of noise does to a field: it hides real differences between models and rewards models that copy annotator habits (Northcutt et al., 2021). It can also reorder model rankings, as shown when TACRED was relabeled (Stoica et al., 2020). Compression has never had that audit.

We propose to build one, and to argue for a specific detection unit. In deletion-based compression, a label error is a wrong decision about a whole grammatical unit: a verb dropped while its object stays, or a modifier kept after its head was cut. A single mislabeled word is the wrong thing to look for. We therefore score each dependency subtree (a word together with everything that grammatically depends on it) by how plausible its annotated keep/delete pattern is under a model that reasons jointly over the tree. We then triage each flagged pair into one of four causes: a real annotation slip, an abstractive rewrite that was forced into deletion labels, a parser mistake, or a genuinely ambiguous case. With the cleaned test sets this produces, we ask a concrete question: is the long-standing finding that syntax-free neural compressors match syntax-constrained ones partly an artifact of noisy references?

Core Claim

Existing label-error detectors for sequences score tokens independently and then aggregate them. Wang and Mueller (2022) found that ranking a sentence by its single worst token beats averaging on CoNLL-2003 named-entity labels (AUPRC 0.4357 versus 0.3022). Compression errors have a different shape.

A head-deletion error spreads low confidence across every word in the affected subtree. Scoring only the worst token keeps one of those signals and averaging dilutes them. More importantly, the error often lies in the combination of decisions. Each individual keep or delete can look reasonable while the joint pattern (modifier kept, head dropped) is implausible. This non-local dependence is what Clarke and Lapata (2008) encoded as hard constraints for generating compressions. We reuse it as a soft signal for auditing them.

The claim has two parts, and both can fail. First, joint subtree scoring finds real compression label errors better than worst-token scoring and better than hard grammar-rule filters. Second, removing or correcting those errors changes how compressors are measured and ranked.

Research Program

Projecting references to labels. We first project each source–compression pair onto per-word keep/delete labels using monotone alignment. Words in the compression that do not align to the source, and reorderings, are recorded as abstractive residue. Toutanova et al. (2016) found that 65.6% of human compressions include substitutions or rephrasing. Gupta et al. describe "stealth" edits such as pronoun resolution. Both mean that some apparent deletion errors are really projection artifacts, and the residue record lets us identify them.

Out-of-sample posteriors. We train a keep/delete model with a tree-structured output layer: a conditional random field over the dependency tree on top of a pretrained encoder. Following the cross-validation recipe of confident learning (Northcutt et al., 2021), we train it in K folds so that every pair is scored by a model that never saw it.

Subtree score. For subtree $T$ with annotated labels $y_T$, source sentence $x$, and parse $\pi$, the score is

$$s(T) = \log p(y_T \mid x, \pi) - \max_{z_T} \log p(z_T \mid x, \pi),$$

where $z_T$ ranges over all keep/delete patterns for $T$. Subtracting the best pattern's log-probability makes scores comparable across subtrees of different sizes. The sentence-level score is the minimum of $s(T)$ over all subtrees in the sentence, which is the tree analogue of worst-token ranking.

Triage. Each flagged pair is sorted by cause:

  1. Abstractive projection noise: the flagged subtree overlaps abstractive residue.
  2. Parser noise: the flag disappears when the same pair is scored under a second, independently trained dependency parser. We will report how often this happens.
  3. Ambiguous hard case: adjudicators disagree about it. This matters because Beigman and Beigman Klebanov (2009) showed that noise from hard cases harms learners differently from random slips, so hard cases should be kept but marked rather than "fixed."
  4. Annotation slip: everything that remains.

Corpora and audit. We apply the pipeline to Ziff-Davis (1,055 pairs), Clarke–Lapata broadcast news (1,070), the Written News Compression corpus (1,629), and the roughly 2-million-pair news deletion corpus of Filippova et al. (2015). We then release audited test splits in the spirit of Re-OIE2016 and Re-TACRED.

For detection ground truth, three trained adjudicators label a stratified sample from the three small corpora. The sample mixes pairs flagged by each detector with random unflagged pairs. For each flagged subtree, adjudicators choose among error, acceptable variation, and ambiguous.

The multi-reference corpus of Toutanova et al. adds a second check that needs no new annotation. Restricted to its near-extractive references, a decision flagged in one reference should disagree with the majority of the other annotators' references more often than an unflagged decision does.

Baselines

Do-nothing reference. Detection is compared against random ranking. Downstream, the reference is the uncleaned benchmark as published. Every detector is also reported as lift over the dataset's base error rate, which the adjudicated sample estimates.

Worst-token confident learning. This is the strongest existing detector: the worst-token self-confidence method of Wang and Mueller (2022), applied to the same keep/delete labels. It gets the same cross-validated posteriors from the same encoder, differing only in using an independent per-token output layer instead of the tree CRF, and the same tuning budget.

Its best reported figure is on a different task, AUPRC 0.4357 on CoNLL-2003. No reported result was found on Lacuna for label-error detection on any compression corpus, so we report it fresh. Beating it means a higher AUPRC on the adjudicated compression errors, with a bootstrap confidence interval that excludes zero difference.

Grammar-policy filter. A rule-based alternative flags any pair that violates Gupta et al.'s eight grammar policies over the same parses. It needs no model. The subtree detector beats it if it matches its recall at clearly higher precision.

Grouped-token ablation. This isolates the key ingredient. It uses the per-token model but takes the minimum token score within each subtree, with no joint model. If this ablation matches the full method, the gain comes from grouping words by syntax, not from joint reasoning, and we will say so.

Neighborhood filter. A kNN filter in encoder space in the manner of LNIC (Desmond et al., 2020) is layered on top of each detector. It tests whether generic false-positive pruning closes the gap.

Compressors for re-evaluation. Three systems are re-scored on original versus audited test sets:

  1. Constrained search: RCGF from Yao and Wan (2017), with the original ILP as reference. Reported grammatical-relation F1 on written news is 66.4 for RCGF and 66.5 for ILP.
  2. Syntax-free neural deletion: a Filippova-style LSTM, which reported Accuracy@1 of 0.30 against 0.21 for a MIRA baseline.
  3. Learned plausibility filter: a CUPS-style plausibility-plus-salience compressor (Desai et al., 2020).

Why not just…?

Why not just drop every pair that violates Gupta's grammar policies? Those policies fire on 59–72% of pairs for a single rule. Most of those firings are abstractive rewrites or legitimate stylistic choices, not errors. A filter this blunt would delete most of the benchmark. It would also bake in exactly the rigid view of grammar that human compressions ignore. The policy-filter baseline measures this directly. If its precision on adjudicated errors turns out comparable to ours, the learned detector is unnecessary and we will report that.

Why not just use worst-token cleanlab on keep/delete tags? This is the most serious threat, because it is cheap and has worked on other token tasks. Our reason for expecting it to fall short is structural. Its score reflects one word, while the characteristic compression error is an inconsistent pattern across a head and its dependents. Yoshikawa et al. (2012) found such patterns matter most when arguments sit far from their predicates. The grouped-token ablation and the head-to-head AUPRC comparison settle this question. A small or null gap would falsify the first half of our claim.

Why not just evaluate with multiple references instead of cleaning? Toutanova et al.'s multi-reference metric correlates with human meaning judgments at about 0.59, but with grammaticality only at about 0.35. The three single-reference corpora cannot be re-collected cheaply, and multi-reference scoring does nothing for training data. We use multi-reference data as a validation signal, not as a replacement for cleaning.

Why not trust scale, since a syntax-free LSTM already reached 4.5/5 readability? Filippova et al.'s result is exactly the one we want to re-test. Noisy references can make a high-capacity model look good for matching annotator habits (Northcutt et al., 2021). Our re-ranking experiment asks whether the LSTM's edge over constrained systems holds up once those references are cleaned.

Decisive Checks

The first check is detection. On the adjudicated sample, the subtree detector must beat worst-token scoring, the grouped-token ablation, and the policy filter on AUPRC and on lift at the estimated number of errors. The triage must also show that parser-unstable flags are a minority. If they are not, the audit is measuring parsers, not annotators.

The second check is evaluation. We re-score all compressors on the audited test splits. We ask whether the gap between constrained and syntax-free systems grows, shrinks, or reverses, and we report Kendall rank changes with bootstrap intervals.

The third check is training. We retrain the LSTM and RCGF's learned weights on cleaned versus noisy training splits of the Filippova corpus. Before any cleaning, we set aside an audited held-out set, and we match training-set sizes by randomly removing an equal number of pairs from the noisy split. We collect blind human grammaticality ratings on this held-out set. This tests whether hard constraints protect a model against training noise, the unresolved question of whether syntax acts as a regularizer.

A null result on the second and third checks would be informative too: it would mean compression benchmark noise, though large, does not distort conclusions. That would be a useful thing for the field to know.