When Cheap Syntax Lets Stereotypes Back In

Coreference resolution is the task of deciding which earlier noun a pronoun refers to. Systems that do it make gendered mistakes: given "The secretary called the carpenter because he was late," they tend to link "he" to "carpenter" even when the sentence's grammar says otherwise. A common hope is that grammatical structure acts as a guardrail. When the sentence structure fixes the answer, the model should follow the structure rather than its stereotype. Recovering that structure exactly means running a full dependency parser, a program that builds a tree of which word grammatically governs which. Full parsers are slow. Engineers therefore increasingly swap them for cheap stand-ins: the distance in tokens between two words, or small linear "probes" that read approximate syntax out of a pretrained model's hidden states. The stand-ins are 100 to 1,000 times cheaper and match the parser on standard accuracy benchmarks. No one has asked whether they keep the guardrail.

This note proposes to measure how stereotype leakage in coreference changes as the structural signal gets cheaper, holding sentence length fixed. It also tests a specific remedy: run the full parser only on the small set of sentences where the cheap signal is unreliable and the decision lies between two occupation nouns. That set is chosen without looking at the pronoun's gender. The claim to test is that most of the fairness benefit of explicit syntax can be bought at a few percent of its cost, and that uniform cheap proxies throw that benefit away exactly where it matters.

Core Claim

The stereotype gap on syntactically determined coreference is not fixed. It grows when the structural signal fails on the arc linking the pronoun to its true antecedent, and that failure is concentrated in a predictable subset of sentences: those where a distractor occupation sits linearly closer to the pronoun than the grammatical antecedent does. If this holds, three things follow. A relative-distance proxy should show a larger pro-versus-anti-stereotype gap than a full parse, but only on distractor-closer sentences. Parse errors on that single arc should statistically account for much of the excess gap. A gender-blind router that sends only uncertain occupation-versus-occupation decisions to a full parser should recover most of the full parser's fairness at a small fraction of its latency.

Motivation and Prior Work

Two results motivate the proposal. Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods built WinoBias, a test set split into Type 1 sentences that need world knowledge and Type 2 sentences whose answer is fixed by syntax. Its end-to-end neural model scored 74.9 F1 on pro-stereotypical and 47.7 on anti-stereotypical Type 1 sentences, a 27.2-point gap. It still leaked stereotypes on Type 2, where syntax should have settled the question. That residual Type 2 leakage is the phenomenon this proposal tries to explain. Gender Bias in Coreference Resolution built Winogender minimal pairs. It found that a rule-based system with hard syntactic sieves resolved 68% of pronoun-swapped pairs differently, and its accuracy fell from 38.3% to 10.0% on "gotcha" cases. So having syntax is not sufficient either. What matters is whether the right arc is recovered and actually used, which is the variable this proposal isolates.

One result constrains the design. Is there Gender Bias in Dependency Parsing? Revisiting "Women's Syntactic Resilience" showed that earlier claims of author-gender bias in parsers (Women's Syntactic Resilience and Men's Grammatical Luck) were driven by sentence length. Any study linking parse quality to bias must therefore separate structural difficulty from length, or it will repeat that error. For this reason the diagnostic below holds length fixed and varies only where the distractor sits.

Two papers supply the cheap structural signals under test. Structural Bias for Aspect Sentiment Triplet Extraction replaces dependency distance with relative token distance injected into attention through a small adapter, reporting about 1,000x faster distance computation and 1,000x fewer added parameters. Soft Syntactic Reinforcement for Neural Event Extraction trains linear probes on the Penn Treebank to bring out latent tree depth and distance inside a pretrained model, with no parser at inference. Neither paper measures bias. Neither is evaluated on sentences where linear distance and syntactic distance disagree, which is exactly where a stereotype prior could fill the gap. The parsing-cost literature makes the cost side concrete. Parsing took more than half of total processing time in Scaling up Biomedical Event Extraction to the Entire PubMed, and query-time parsing latency was unacceptable in System Demonstration of On-Demand Information Extraction. This is why "just parse everything" is not a free answer.

Finally, Directional Bias Amplification supplies the measurement lens. It separates attribute-to-task from task-to-attribute bias and warns that miscalibrated thresholds can look like amplification. The proposal reports both directions and calibrates every system before comparing them.

Research Program

The first component is a controlled diagnostic set. It extends WinoBias Type 2 templates using the same 40 occupations, with gold dependency trees known by construction. For each base sentence we generate length-matched variants that differ only in arrangement. In the "antecedent-closer" arrangement the grammatical antecedent is the linearly nearest occupation. In the "distractor-closer" arrangement a relative clause or passive construction places the other occupation between the antecedent and the pronoun. Variants are matched in token count, and in tree depth where possible, so that sentence length cannot explain any difference, following the lesson of the dependency-parsing replication. Gender is crossed fully with arrangement, so pro- and anti-stereotypical items appear equally in every cell. A second, harder slice adds one level of embedding at fixed length to stress the probes and the parser.

The second component is a ladder of coreference systems that share one pretrained encoder and one coreference head. They differ only in their structural input. The rungs are: no structural input; the relative-distance adapter; the linear syntactic probes; predicted trees from a graph-based dependency parser fed in as attention bias; and gold trees, which serve as an upper bound. Every rung is trained on the same OntoNotes 5.0 data with the same tuning budget. Each rung is run with and without gender-swapped data augmentation, because augmentation is the strongest known debiasing step and the structural effect must survive it to matter.

The third component is the selective parser. At inference, the cheapest rung (the distance adapter) proposes antecedents. A sentence goes to the full parser when two conditions hold: the top two candidates are both occupation mentions, and either the margin between them is small or the adapter's nearest-candidate choice disagrees with the probe-derived syntactic head. The router may not see pronoun gender or any gendered word. We check this directly: routing decisions must be identical across each gender-swapped minimal pair, otherwise the router itself becomes a channel for bias. The routing threshold sweeps out a curve of cost against fairness.

The analysis component is a mediation test. For each system and sentence we record whether the predicted structure contains the correct pronoun-to-antecedent arc. We then ask how much of the excess anti-stereotype error on distractor-closer items disappears once we condition on that arc being correct.

Baselines

The naive reference point is the plain encoder with no structural input, trained on the same data. It shows what implicit syntax alone gives, and it answers the reviewer view that pretrained models already encode syntax. The comparison that matters is its gap on distractor-closer versus antecedent-closer items at matched length. Any structured system must reduce that difference to count as an improvement.

The strongest existing debiasing method is the recipe from the WinoBias paper: gender-swapped data augmentation combined with debiased embeddings. It reduced the Type 1 gap from 27.2 to 2.1 F1 points, and on SoWinoBias's gender-neutral "they" items the same family of methods brought the gap to 2.1% (Second Order WinoBias). We found no reported Type 2 number for this recipe on Lacuna. We apply augmentation to every rung, so the baseline is "augmentation without structure" and the treatments are "augmentation plus structure," trained on the same data, with the same number of tuning trials and the same encoder. Beating this baseline means a statistically reliable reduction in the distractor-closer gap, measured with paired tests over minimal pairs, with OntoNotes F1 no more than one point lower.

The strongest existing cheap structural methods are the relative-distance adapter and the syntactic probes. We reimplement both on the same encoder. We found no reported result on Lacuna for either method on WinoBias, Winogender, or any bias benchmark. Their reported numbers are on aspect-sentiment extraction and event extraction (for example, MUC-4 recall of 67.75% for the probe method), which is a different task. The selective parser beats them if it closes a substantial share of the gap between the cheap proxy and the full parse on distractor-closer items while parsing only a small fraction of sentences. The yardstick is how much of the full parser's gap reduction it recovers per unit of added latency.

The full-parse rung is the expensive ceiling. The selective method does not need to beat it on fairness. It needs to approach it at a fraction of the wall-clock cost, measured on the same hardware.

The key ablation is a random router that parses the same fraction of sentences, chosen uniformly at random. If random routing at an equal budget does as well as uncertainty routing, then the routing signal adds nothing and the claim fails. A second ablation removes the requirement that both candidates be occupations, which tests whether the benefit comes from targeting demographically loaded decisions specifically.

Why not just…?

Why not just use a large pretrained model and trust its implicit syntax? Both structural papers report that explicit structure still improves strong pretrained encoders on relational extraction. FairBelief found that a larger model (VICUNA-7b, HONEST 0.257 ± 0.038) was more hurtful than DistilBERT (0.017 ± 0.020), so scale does not reliably remove stereotypes. Neither result is a coreference fairness result, though. The no-structure rung settles the question directly. If it shows no distractor-closer excess, the proposal's premise is wrong and we will say so.

Why not just apply gender-swap augmentation and stop? This is the real threat. Augmentation removed most of the Type 1 gap, and it may also remove the Type 2 residual without any syntax. The two-by-ladder design is built to settle this. If every structured rung with augmentation matches unstructured augmentation on distractor-closer items, then structural fidelity does not matter once the data is balanced, and we report that negative result. Augmentation does not fix the arc error itself, so we expect a residual on distractor-closer items. That is a prediction to be tested, not an assumption.

Why not just calibrate outputs or apply corpus-level constraints after the fact? Threshold tuning reduced measured amplification on COCO from 0.040 to 0.001 in Directional Bias Amplification. The corpus-level constraint method of Men Also Like Shopping cut amplification by 40.5% on visual role labeling, but it needs the whole test batch at once. Calibration adjusts aggregate rates. It cannot make an individual sentence resolve to its grammatically correct antecedent, and per-sentence correctness is what WinoBias-style pairs measure. We still apply calibration to every system before comparison, so that any remaining difference cannot be a threshold artifact.

Why not just parse everything with the full parser? On this small benchmark that is affordable. It is the ceiling rung. The costs reported in the PubMed and on-demand extraction papers show why it is not affordable in deployed pipelines, which is why the cost-fairness curve rather than the ceiling is the deliverable.

Proposed Experiments and Decisive Checks

The first check tests the mechanism. On the length-matched diagnostic, compare the pro-versus-anti gap of the relative-distance adapter and the full parse in the antecedent-closer and distractor-closer cells. The hypothesis predicts an interaction: the proxy's gap is elevated only when the distractor is closer. A uniformly elevated gap would instead point to a general capacity difference.

The second check is mediation. The test is whether conditioning on correct recovery of the pronoun-antecedent arc removes most of the excess anti-stereotype error. If it does not, structural fidelity is not the channel, even if the gaps differ.

The third check transfers the findings to existing benchmarks: WinoBias Type 1 and Type 2, Winogender, and SoWinoBias, all trained on OntoNotes. Report task F1, the pro/anti gap, and both directional amplification measures. Results should hold on the Type 2 slice and need not hold on Type 1, where syntax does not settle the answer. Improvement on Type 1 would suggest something other than syntax is at work.

The fourth check is the cost curve. Plot measured per-sentence latency against the distractor-closer gap for every rung and for the selective parser across routing thresholds, alongside the random-routing ablation at matched budgets. Verify that routing decisions are invariant under gender swap.

The proposal succeeds if uncertainty routing that sends a small minority of sentences to the parser recovers most of the full parser's gap reduction, and random routing at the same budget does not. It fails cleanly if augmentation alone erases the distractor-closer interaction.