Preservation as the Default: Back-off Hierarchies That Know When Not to Disambiguate

Many sentences can be read more than one way. In "we meet the colleagues in Berlin", the phrase "in Berlin" may say where the meeting happens or which colleagues are meant. Translation systems usually settle every such question before translating, which means they will sometimes choose wrong. The alternative is a safe translation, a target sentence that stays true under all the source readings and leaves the choice to the reader. Classical systems showed this is possible when the target language has a matching ambiguity, but they never answered when a system should preserve and when it should commit. This proposal answers that question with a structure from the rule-based tradition. A back-off hierarchy tries narrow, specific evidence first and falls back to broader evidence. We change two things about it. The bottom of the hierarchy is "preserve the ambiguity" rather than a default guess. And a hierarchy level that has plenty of data but shows a split between readings counts as positive evidence that the sentence is genuinely ambiguous, not as a reason to fall back to a coarser level.

In two sentences: specific-over-general rule precedence decides whether to disambiguate, not just which reading to pick. It lets a translator tell "I lack evidence" apart from "the evidence says both readings are fine", something a single confidence score blurs together. We test this on prepositional-phrase attachment, where the classic disambiguation benchmarks give reported numbers to beat. We add a new annotated layer to those benchmarks that marks which items are genuinely ambiguous, and we evaluate English–German translation, where attachment ambiguity can often be carried over.

Where the two directions meet, and what is missing

Rule-based and hybrid translation systems resolve collisions between rules by specificity: the rule with more lexical content or tighter constraints wins over the generic one. Takeda's pattern grammars do this by preferring rules dense in terminal symbols (Pattern-Based Context-Free Grammars for Machine Translation). Khalilov et al. apply fully lexicalized reordering rules before part-of-speech rules (A New Subtree-Transfer Approach to Syntax-Based Reordering). Xia and McCord order learned rewrite patterns into a strict specific-overrides-general hierarchy (Improving a Statistical MT System with Automatically Learned Rewrite Patterns). These papers motivate the proposal: flat rule sets collide destructively, as Zhu and Wang's sudden quality drops under rule additions show (The Effect of Adding Rules into the Rule-based MT System). All of them share one limitation the file names directly. Specificity is static. A highly specific rule fires whether its evidence is strong or weak, and it always outputs a single decision.

Collins and Brooks' backed-off attachment model is the statistical form of the same idea and supplies our method (Prepositional Phrase Attachment through a Backed-off Model). It consults counts for the full word tuple (verb, object noun, preposition, noun inside the prepositional phrase), then progressively smaller sub-tuples, and at the bottom falls back to noun attachment. It always outputs one attachment.

The ambiguity-preservation line supplies the target-side machinery. Emele and Dorna transfer packed representations, which keep alternative analyses as local disjunctions in one shared structure, and they resolve only those distinctions the target cannot express (Ambiguity Preserving Machine Translation using Packed Representations). Dorna et al. do the same over underspecified semantic structures (Syntactic and Semantic Transfer with F-Structures). Knight and Langkilde intersect the generation forests of each reading to find strings valid under all of them (Preserving Ambiguities in Generation via Automata Intersection). These are the closest prior work, and they differ from this proposal in a precise way. Emele and Dorna's override from preserving to disambiguating rules is triggered only by the target language's limits, and those rules are written by hand. Knight and Langkilde treat every reading as a hard constraint and only gesture at a weighted fallback when intersection gives an empty or awkward result. Neither asks whether the source evidence justifies committing. None of these papers reports a quantitative evaluation.

Core Claim

A back-off hierarchy over source evidence can make a three-way decision per ambiguous site: attach to the verb, attach to the noun, or preserve. Preservation sits at the bottom as the default. A decisive specific level overrides it, as in any specificity system. The new element is how a level is read. For the most specific level $\ell$ whose context count $c_\ell$ reaches a reliability threshold $k$, let $p_\ell$ be the observed fraction of verb attachments. The decision is

$$ d = \begin{cases} \text{resolve to majority reading} & \text{if } |p_\ell - \tfrac12| > \delta(c_\ell) \\ \text{preserve} & \text{otherwise,} \end{cases} $$

where $\delta(c_\ell)$ shrinks with $c_\ell$, like a binomial confidence half-width. If no level reaches $k$, the default is also preserve, not noun attachment.

This rule separates two situations that a flat classifier's probability of 0.5 conflates. In one, the specific context is too sparse to judge. In the other, the specific context is well attested and both attachments really occur, like Hindle and Rooth's "signed a settlement agreement with them", which they describe as systematically ambiguous (Structural Ambiguity and Lexical Relations). The hierarchy also addresses the static-specificity problem the file identifies, because a specific rule wins only when its counts support it.

A second element on the target side sets the cost of preserving. When the target language offers a matching ambiguous construction, as German often does for attachment, preservation is cheap. When it does not, preserving costs fluency, and the threshold $\delta$ is lowered so the system commits more readily. This makes Emele and Dorna's target-triggered local disambiguation automatic and connects it to source confidence.

Research Program

We would build three components.

The first is the hierarchical decision model. It uses Collins and Brooks' back-off levels, trained on their 20,801-quintuple training split. The thresholds $k$ and $\delta$ are tuned on the 4,039-item development split against a new annotation layer, described next.

The second is that annotation layer, which addresses Gap 4 in the file: there is no benchmark for preservation. For roughly 1,500 items drawn from the Wall Street Journal development and test splits, we recover the full sentences. Three annotators label each item verb-attached, noun-attached, or "both readings acceptable in context". Collins and Brooks' reported human ceiling of 88.2% with head words alone suggests a non-trivial share of items is genuinely open, and Hindle and Rooth's 10–15% human error rate points the same way. Measuring that share is itself a result.

The third is a translation stage for English–German. A neural translation system produces an n-best list. A German parser, with human verification on a sample, labels each candidate as target-ambiguous, verb-attached-only, or noun-attached-only. Then:

  • a "preserve" decision selects the most fluent target-ambiguous candidate;
  • a "resolve" decision selects the most fluent candidate expressing the chosen reading;
  • if no target-ambiguous candidate exists, that cost feeds back into $\delta$.

We keep generation inside an off-the-shelf system rather than reimplementing forest intersection, because the question is the decision policy, not the generator.

Baselines

The naive reference is the default-attachment heuristic that sits at the bottom of classical back-off. Brill and Resnik report 64.0% for it on WSJ (A Rule-Based Approach to Prepositional Phrase Attachment Disambiguation). It is run on our identical items and annotation, with the same three-way scoring.

The strongest existing disambiguator is Collins and Brooks' backed-off model, reported at 84.5% attachment accuracy on the WSJ test set. It shares our training data, feature levels and counts exactly, so it differs from our system only in its decision rule and bottom default. To count as beating it, our method must lower error on the items it resolves at matched coverage. It must also produce translations that humans judge correct under the annotated reading(s) more often. The key test is on items labeled "both acceptable", where any forced resolution discards a valid reading. Brill and Resnik's transformation learner, at 81.8% with WordNet classes, is run as a second rule-based disambiguator with the same training split.

The strongest preservation baseline is "always preserve when the target allows", in the spirit of Knight and Langkilde and Emele and Dorna. For it we found no reported result on Lacuna, since neither paper reports numbers. It uses the same n-best lists and candidate labels. It shows the fluency and misdirection cost of unconditional preservation.

The ablation that isolates the key ingredient is a flat, calibrated attachment classifier that abstains to "preserve" when its probability falls within a tuned band around 0.5. It is trained on the same data, and its band is tuned with the same development budget as our $k$ and $\delta$. It keeps everything except the level-wise separation of sparse evidence from balanced evidence. A second ablation keeps the hierarchy but restores noun attachment as the bottom default, which isolates the preserve-as-default choice.

Why not just…?

Why not just disambiguate everything with the best available model? Every forced choice loses on genuinely ambiguous items, whatever the model's accuracy. Hindle and Rooth document systematic ambiguity where both attachments are true. MuCoW shows that translation systems' sense choices degrade on out-of-domain senses (The MuCoW Test Suite at WMT 2019). The experiment settles this directly. If the "both acceptable" share turns out to be tiny, the proposal's value shrinks to a narrow selective-accuracy gain, and we would report that plainly.

Why not just threshold a single classifier's confidence and preserve when it is unsure? This is the real threat. Hindle and Rooth already show that restricting to high-confidence decisions drops their error from 22% to 15%, which is selective prediction without any hierarchy. Our claim is narrower and testable. A flat score cannot tell a sparse context from a well-attested balanced one, so its abstentions should mix cases where the reader faces genuine ambiguity with cases where the system simply lacks data. We compare how well each method's "preserve" signal identifies annotated genuinely-ambiguous items, using area under the ROC curve. If the flat classifier matches the hierarchy, the specificity contribution is refuted, and the proposal reduces to preserve-as-default plus confidence abstention.

Why not just always produce a safe translation? Knight and Langkilde themselves report awkward outputs such as "John's car's keys". They also note that preserved strings can still steer readers toward one reading. The always-preserve baseline measures this cost in human fluency and reading-bias judgments. Our method should match its coverage of genuine ambiguity while avoiding its fluency losses on items where evidence is decisive.

Why not just train a multi-label neural translator such as SCONES? SCONES, which scores several target tokens as independently valid and reports 1–9% relative BLEU gains on WMT19 (Jam or Cream First?), still decodes one sentence. It provides no check that the sentence keeps a structural ambiguity. It could serve as our n-best generator, but it does not replace the decision.

Proposed Experiments and Decisive Checks

The first check is intrinsic. On the annotated WSJ layer we report:

  • three-way decision accuracy;
  • selective error on resolved items, plotted against coverage for all systems;
  • the ROC area comparison between the hierarchy and the flat abstaining classifier for detecting "both acceptable" items.

We also report, as a descriptive result, the share of items annotators judge genuinely ambiguous, with inter-annotator agreement.

The second check is downstream English–German translation of the full sentences. Bilingual judges rate two things for each output: whether it is correct under every annotated acceptable reading, and whether it is fluent. We compare all baselines, with significance tests by item-level bootstrap.

A robustness probe repeats the intrinsic check on Hindle and Rooth-style news text. There, domain shift should make specific levels sparser, and the hierarchy should preserve more rather than guess more.

Three outcomes would decide the matter:

  • The proposal is supported if the hierarchy beats the flat abstaining classifier at detecting genuine ambiguity. It must also beat Collins and Brooks on correctness-under-all-readings without losing on fluency relative to always-preserve.
  • The hierarchy claim fails if the flat classifier matches it on genuine-ambiguity detection.
  • The preservation program is weak on this phenomenon if genuinely ambiguous items are rare.