Teaching Distantly Supervised Relation Extractors to Read the Path Between Entities

Relation extraction systems learn to tag sentences with facts like "X was born in Y." They are usually trained on distant supervision: labels made automatically by assuming that any sentence mentioning two entities that a knowledge base links expresses that link. These labels are cheap and very noisy. A sentence about Obama's foreign policy that happens to mention the United States gets labeled born_in. The usual fixes hide the entity names, but they leave every other word in the sentence untouched. We propose a different fix, borrowed from psycholinguistic tests of grammar. Keep the words on the grammatical path that connects the two entities, and replace every other content word with a nonsense word of the same part of speech and inflection, a "Jabberwocky" substitution. Separately, generate automatic contrast pairs in which the relation word is still present but no longer grammatically links the two entities. The first operation tests whether a model reads the structure that binds the entities or just the sentence's topic words. The second turns that test into free training data that punishes topic-matching.

The opportunity is concrete. The strongest recent work on shortcuts in relation extraction removes entity-level shortcuts and explicitly leaves "syntactic or structural biases" to future work (Rethinking Relation Extraction). The grammar-probing literature, meanwhile, has strong tools for separating structure from meaning, but has only applied them to agreement and parsing probes on frozen models (Gulordava et al., Maudslay et al., Arps et al.). Bringing those tools into the training-data pipeline of distant supervision, as both a diagnostic and a data generator, has not been done in the material we surveyed.

Core Claim

Distantly supervised relation extractors get much of their accuracy from topic words that lie off the grammatical path between the two entities. That is exactly where distant-supervision false positives carry their spurious signal. Two automatically generated, syntax-controlled perturbations fix this better than entity masking, dependency-path pruning, or generic consistency training. The first is path-preserving Jabberwocky, which strips the meaning from off-path content words while keeping every function word, inflection, word position and tree shape. The second is binding-breaking contrast pairs, which keep the trigger word but re-attach it so it no longer connects the entity pair. We make three testable predictions. Standard models will lose much more accuracy on path-preserving Jabberwocky than on equally aggressive substitutions applied on the path. Training with the contrast pairs will shrink that gap. Training with the contrast pairs will also improve accuracy on entity-debiased evaluation.

A hard constraint shapes the design. Full Jabberwocky, as in Maudslay et al., cannot be applied to relation extraction. The relation is carried by content words like "married" or "hometown," so nonce-ifying everything erases the label itself. Isolation must therefore be selective: the meaning on the path stays, and the meaning off the path goes. This is what separates the proposal from simply porting the probing method.

Research Program

The first component is a stress-test construction. For each labeled sentence, we parse with two dependency parsers and find the shortest dependency path between the subject and object spans, a parser-derived chain of head-dependent links. We keep only sentences where both parsers agree on that path, to limit errors introduced by parsing. Off-path content words are then replaced in two strengths. The milder strength follows Gulordava et al. and uses real words matching part of speech and morphological features. The stricter strength uses pronounceable pseudowords in the style of Maudslay et al.. Entities are masked with coarse types, as in Zhang et al., so entity and context shortcuts are not confounded.

Every perturbed sentence gets a matched control. The control applies the same number of substitutions, but to on-path content words. The gap between off-path and on-path damage is the quantity of interest. If a model reads binding structure, off-path substitution should barely hurt it, while on-path substitution should hurt it a lot. A topic-matching model shows the opposite pattern or no difference. Because roughly 78.7% of TACRED is no_relation (Zhang et al.), we report positive-class F1 and the false-positive rate on no_relation items separately.

The second component generates binding-breaking contrast pairs. The source is a distantly supervised positive whose trigger word lies on the path. We make a rewrite that keeps the trigger in the sentence but breaks the link: one entity is moved into a clause, or swapped with a third noun phrase, so that the parse no longer connects it to the trigger. The rewrite is labeled no_relation. Rewrites are produced by templated tree operations and filtered by parser agreement. A small, separately audited sample estimates how often the automatic negative label is wrong. This is distant supervision whose labeling rule is grammatical rather than co-occurrence-based. It is also the clearest contrast with Rethinking Relation Extraction, which swaps entities but keeps contexts fluent and structurally unchanged.

The third component is training. We fine-tune a pretrained encoder on distant-supervision data, adding path-preserving Jabberwocky copies of positives (label kept) and binding-breaking pairs (label flipped). The augmented copies are mixed in at a fixed ratio, tuned on a development set.

Baselines

The naive reference point is the same encoder fine-tuned on raw distant-supervision data with coarse entity-type masking and nothing else. Every gain is measured against it.

The strongest existing method on shortcut robustness is MixDebias from Rethinking Relation Extraction. It combines entity-dictionary replacement with counterfactual subtraction of entity-only predictions, and it raised LUKE's F1 on DREB-TACRED from 44.40 to 62.44. It is the right competitor because it targets the same failure, models predicting relations without reading the context, and it is the published state of the art on DREB in our file. We will reimplement it on the same encoder and training data, with the same tuning budget in development-set trials and the same number of training steps. Beating it means exceeding its DREB F1 under our reimplementation. It also means a clear advantage on the path-preserving Jabberwocky set, without a loss on standard TACRED or Re-TACRED test F1. We also test combining the two methods, since they target different shortcuts.

For the distant-supervision regime on NYT, the strongest prior number in the file is 55.4 F1 from multi-task training with ACE05 (Distantly Supervised Entity Relation Extraction with Adapted Manual Annotations). That system uses clean manual data, which ours does not. We therefore report it as a reference point, and we also run our augmentation on top of it to see whether the gains add up.

Two denoising baselines represent the noise-handling tradition. One is the Mean Teacher consistency method (Semi-Supervised Teacher-Student Architecture for Relation Extraction), run with its standard random perturbations. The other is entity blanking (Xiao et al.). Neither has a reported number on our benchmarks in the file. The one most structurally similar to ours is a shortest-dependency-path model that feeds only path tokens, the neural descendant of the syntactic features in Mintz et al..

Three ablations isolate the key ingredient. The first applies random-token off-path substitution that ignores part of speech and morphology, testing whether preserving grammar matters. The second applies on-path-matched substitution in training, testing whether the location of the perturbation matters. The third uses Jabberwocky copies without binding-breaking pairs, and the reverse. Each ablation uses the same mixing ratio and step budget.

Why not just…?

Why not just mask the entities, as everyone already does? Entity masking removes one shortcut and leaves the rest of the sentence free to act as a topic cue. Rethinking Relation Extraction reports 30% to 50% F1 drops for strong models once entity shortcuts are broken. Maudslay et al. show that context-word meaning alone props up apparent syntactic competence, which drops 15.4 UUAS points under Jabberwocky. Our naive baseline already includes entity masking. If it shows no off-path versus on-path gap on the stress set, this objection wins and the proposal's premise fails.

Why not just feed the model only the shortest dependency path? This is the most serious threat, because it makes off-path words invisible by construction. Two costs argue against it. First, pruning throws away information: Mintz et al. found that syntactic features alone gave lower recall than syntactic plus lexical features. Second, pruning makes every prediction hostage to a single parse at test time. Our method uses the parser only when generating training data, and at test time the model reads the full sentence. Pruning could still match us on robustness. The comparison settles this on two fronts: test F1 on standard sets, where pruning should lose recall, and the stress set, where pruning is immune by design. If pruning matches us on both, we will report that and recommend it.

Why not just add a little clean manual data? Clean labels fix label noise but not shortcuts. In Rethinking Relation Extraction, models trained on human-annotated TACRED still fall to 44.40 F1 on debiased evaluation. Distantly Supervised Entity Relation Extraction with Adapted Manual Annotations reports that naively merging manual and distant data hurt. The stacking experiment on NYT tests whether the two help in complementary ways.

Why not just use consistency regularization? Generic perturbation consistency rewards stable predictions, including predictions that are stably wrong because of topic cues. With an estimated 61.8% false positives among document-level distant-supervision relations (Xiao et al.), consistency can lock in noise. The Mean Teacher baseline uses identical compute. If it closes the off-path gap as well as our method does, then grammar-aware targeting adds nothing.

Proposed Experiments

The main supervised experiment trains on TACRED and evaluates on four sets: TACRED test, Re-TACRED, DREB-TACRED, and our path-preserving Jabberwocky version of TACRED test at both substitution strengths. Positive-class micro-F1 is the main metric; the off-path versus on-path gap is the diagnostic. The reference numbers are 65.4 F1 on TACRED for the position-aware LSTM (Zhang et al.) and LUKE's 44.40 and 62.44 on DREB-TACRED. We found no reported result in the file for Re-TACRED or TACREV.

The distant-supervision experiment trains on NYT and evaluates on its test set and a Jabberwocky version of it. It adds a manually audited sample of about 500 training instances, used to check a secondary use of the diagnostic. In that check, each training instance is scored by how much the model's confidence drops under off-path Jabberwocky, and we test whether that score predicts audit-confirmed false positives better than raw model confidence does. DocRED, with its 58.43 F1 reference from Xiao et al., is a stretch target. Paths that cross sentences need coreference links, so we treat it as exploratory.

Three outcomes decide the proposal. First, baseline models must show a clearly larger accuracy loss from off-path than from on-path substitution; otherwise the shortcut we target does not exist. Second, binding-breaking pairs must cut the rate of false positives on contrast items without lowering standard-test F1 by more than about a point; otherwise the method trades accuracy for robustness. Third, the method must beat dependency-path pruning on standard-test recall while matching it on the stress set; otherwise the simpler pruning approach is the right answer. We will also measure the label-error rate of automatically generated negatives on an audited sample, because the method is only as good as its grammatical labeling rule.