Role-Aware Copying for Implicit Causal Extraction

When a generative model extracts cause–effect pairs from text, it usually copies the event phrases directly out of the input. A pointer-generator is the standard tool for this: a decoder that, at each step, either writes a word from its vocabulary or points at a word in the input and copies it. The pointer decides what to copy from attention weights, and attention is easily drawn to words that look relevant on the surface. In causal extraction, that weakness has a specific consequence. The model can find the right event phrases and still assign them to the wrong roles. Swap the order of two clauses, drop the "because," or put the sentence in the passive voice, and a model that copies by surface salience will tend to copy the same spans into the same cause and effect slots. Its token-level output stays correct while its causal claim is reversed.

We propose to split the copy decision into two parts. The first asks whether a span is an event worth copying, and relying on surface cues is acceptable here. The second asks which causal role the span plays, and this part must depend on composition and direction. The role part is scored against embeddings of natural-language definitions of "cause" and "effect," so each slot asks what kind of span it wants rather than relying on an arbitrary learned slot index. It is trained with a swap-consistency objective: on an automatically constructed minimal pair in which the roles are inverted, the role score must flip while the event score stays fixed. The claim is that this factorization removes a failure mode that current generative causal extractors do not measure, and that it generalizes to unseen kinds of inversion better than simply adding swapped examples to the training data.

Core Claim

Lexical overlap bias is the habit of treating shared words as evidence of shared meaning. Work on paraphrase and similarity has documented it thoroughly. In PAWS, BERT models above 90% on Quora Question Pairs fall to 33–40% on word-scrambled pairs such as "flights from New York to Florida" versus "from Florida to New York." In Aiming beyond the Obvious, more than half of standard similarity pairs turn out to be "obvious" cases where overlap alone predicts the label. Both papers motivate this proposal: they show that encoders conflate token identity with relational meaning. Neither touches generation or causality.

In generative causal extraction, the same conflation lives inside the copy distribution. SEAG linearizes causal graphs as [Event A, Relation, Event B, SEP] and uses hard constrained decoding to restrict output tokens to the source text and relation labels. It notes that event ordering matters, but it guarantees only that copied tokens exist in the input, not that they occupy the correct role. ICE aligns cause and effect representations with optimal transport inside a BART template generator but never tests role inversion. FiD-PGN is the closest pointer-plus-overlap study. It shows that a cheap pointer reusing decoder cross-attention improves question answering by 9.6 points on answers absent from training. That paper supplies our copy mechanism, but its overlap split concerns answer memorization in factoid QA, not relational direction.

The new element is a pointer whose per-slot copy distribution factorizes as

$$a_i^{(r)} \propto s_i \cdot \exp\big(\langle W h_i,\; e_r \rangle\big),$$

where $a_i^{(r)}$ is the probability of copying input token $i$ into causal slot $r \in \{\text{cause}, \text{effect}\}$, $s_i$ is a role-agnostic event-salience score (the existing cross-attention, as in FiD-PGN), $h_i$ is the contextual encoder state of token $i$, $W$ is a learned projection, and $e_r$ is the encoder embedding of a natural-language definition of role $r$ (for example, "the event that brings about the other event"). The role term is trained with a swap-consistency loss on minimal pairs $(x, \tilde{x})$ in which the cause and effect spans exchange roles:

$$\mathcal{L}_{swap} = \sum_{i \in \text{spans}} \Big\| \log \tfrac{a_i^{(c)}(x)}{a_i^{(e)}(x)} + \log \tfrac{a_i^{(c)}(\tilde{x})}{a_i^{(e)}(\tilde{x})} \Big\|^2 + \lambda \,\| s(x) - s(\tilde{x}) \|^2 ,$$

where superscripts $c$ and $e$ denote the cause and effect slots and $\lambda$ weights the term that keeps event salience invariant under the swap. The first term asks that a span's cause-versus-effect preference reverse sign on the swapped sentence. The second asks that "is this an event" not change.

This factorization also bears on a tension the file identifies between counterfactual debiasing and copying. CF-ECI improves event causality classification by subtracting the score of masked-context and masked-event inputs. This suppresses prominent event words such as "earthquake" that predict causality spuriously. A pointer needs to put high attention on exactly those words to copy them. Placing the debiasing pressure only on the role factor lets the model copy "earthquake" faithfully while preventing its frequency from deciding whether it is the cause. CF-ECI therefore constrains the design: debiasing applied to the whole copy distribution would harm extraction, so we do not apply it there.

Research Program

The work has three parts. The first is a diagnostic test suite. Starting from RECESS, which annotates cause, effect, and signal (connective) spans over the Causal News Corpus, we build four families of controlled variants. Role-swap variants exchange the cause and effect spans around the signal ("A because B" becomes "B because A"). The roles then invert by construction, even when the result is less plausible. Voice and nominalization variants rephrase the relation so that surface order and causal order disagree. Connective-deletion variants remove the signal span, turning an explicit relation into an implicit one where the sentence still reads coherently. Overlap distractors pair non-causal CNC sentences with the same event phrases taken from causal sentences. Human annotators validate a sample of each family. RECESS reports Krippendorff's alpha of only 42.96% overall and 28.45% for signals, so we report agreement on our variants as well and restrict headline numbers to items that pass validation. On this suite we measure a role-inversion rate (the fraction of swapped items where the model copies the original assignment), a distractor false-positive rate, and span F1. Answering the question "do current pointer-based and constrained extractors keep their role assignment under inversion?" is a finding in its own right, whatever happens with the method.

The second part is the model. We use BART-large as a template generator in the SEAG/ICE style, add a FiD-PGN-style pointer that reuses cross-attention, and give it two copy slots that use the factorized distribution above. The role definitions are encoded by the same encoder, so the slot queries live in the input's representation space. Training uses the standard generation loss plus $\mathcal{L}_{swap}$ on swap pairs generated automatically from training data only, using the role-swap family alone. Voice, nominalization, and connective deletion are held out entirely to test generalization.

The third part asks whether using definition embeddings, rather than learned slot vectors, buys anything beyond robustness. We test sensitivity to paraphrased definitions, since the semantic-label literature reports that label wording matters, as Talavera Cepeda et al. note for NLI-style label hypotheses. We also test transfer to relation types not seen in training, using definitions alone, on MAVEN-ERE relation types. One concern is that definition text might itself invite surface matching against the input, as CaKNowLI risks when it verifies causal hypotheses with NLI. Our design limits this exposure because the definition embedding is compared with contextual span states, not concatenated with the input as a hypothesis. A control replacing the definitions with lexically mismatched paraphrases checks whether that matters.

Baselines

The naive reference point is a plain BART-large template generator with a standard single-gate pointer-generator. It is trained on the same data with the same tuning budget, and it shows how often an unconstrained copier inverts roles.

The strongest published generative competitor is SEAG, which uses hard constrained decoding and structural discrimination against negative graphs. On EventStoryLine it reports undirected F1 of 34.85 against 29.46 for the previous best pipeline, 49.92 F1 on MAVEN-ERE, and 91.60 undirected F1 on SCITE (SEAG). No reported directed-setting result for SEAG was found on Lacuna. Directed scoring is where role inversion shows up, so we rerun SEAG with its released configuration on the same backbone and splits and score it both ways. For span extraction on RECESS, the strongest reported system is ALBERT with beam-search span selection and a signal classifier, at 67.69 F1 on cause-effect-signal span detection (RECESS). We retrain it as a non-generative reference. ICE reports 55.02 F1 on ECE-CCKS (ICE), and we include it there. For the classification side of implicit detection, CF-ECI (55.1 F1 on EventStoryLine) and SemSIn (56.1 F1 on EventStoryLine, 58.3 on Causal-TimeBank, SemSIn) give reference points for whether a causal link is detected at all. They do not extract directed spans. All generative systems share the backbone, the training data, the same number of hyperparameter trials, and comparable GPU-hours.

Three ablations isolate the key ingredient. The first removes $\mathcal{L}_{swap}$ but keeps the factorized pointer. The second replaces definition embeddings $e_r$ with randomly initialized learned slot vectors, isolating the semantic label component. The third removes the factorization and applies the swap loss to the whole copy distribution, which tests the claim that debiasing must avoid the salience term. Beating the strongest competitor means two things. In-distribution directed F1 on EventStoryLine and RECESS must match or exceed SEAG and the RECESS baseline within seed variance. The role-inversion rate on the held-out inversion families (voice, nominalization, connective deletion) must also fall clearly below that of every baseline, including the augmentation baseline described next. No reported result for any system on this suite was found on Lacuna, since the suite is new.

Why not just…?

Why not just add role-swapped examples to the training data? This is the most serious threat. In PAWS, adding 12,000 scrambled examples lifted BERT's challenge accuracy from 33% to 85% (PAWS), and the same could plausibly work here. We therefore train the plain pointer baseline and SEAG on exactly the same swapped pairs our model sees, which isolates the architectural contribution from the data contribution. Our bet is that augmentation teaches the specific transformation it was shown, while the factorized loss teaches the abstract invariant, so the gap should appear only on held-out families such as passives and connective deletion. If augmentation matches us there, the factorization is unnecessary and we will report that. One piece of mild evidence against generic augmentation is that RECESS found data augmentation did not help cause-effect-signal span detection and slightly hurt it, although their augmentation was not role-targeted.

Why not just keep SEAG's hard constrained decoding? Hard constraints guarantee that every copied token exists in the input, which addresses hallucination, but they say nothing about which slot a valid token belongs in. A role-swapped sentence passes every vocabulary constraint while producing an inverted graph. Directed scoring of SEAG on the role-swap family shows this directly. If SEAG's structural discrimination already yields low inversion rates, the problem is smaller than we claim.

Why not just apply CF-ECI-style counterfactual subtraction? CF-ECI is a binary classifier over given event pairs. It neither extracts spans nor assigns direction. Applied to the full copy distribution, subtraction would push down exactly the prominent event tokens the extractor must copy. Our third ablation, swap loss on the full distribution, is the closest proxy for this option. A drop in span F1 relative to the factorized version would show the cost.

Why not just prompt a large language model? The evidence in the file is unfavorable. ChatGPT reached 6.87 F1 on ECE-CCKS against ICE's 55.02 (ICE). On COPES-hard, GPT-4 produced so many false positives that a synthetic-control pipeline improved precision by nearly 30% relative to it (Synthetic Control). We include a few-shot GPT-class run on our test suite only as a reference for inversion rate, not as a primary competitor.

Decisive Checks

The first check is diagnostic. If the plain pointer and SEAG already keep their role assignment on validated role-swap items, with inversion rates near the human error rate, the motivating failure does not exist at a scale that matters, and the method half of the project should stop. The second check decides the main claim. On the held-out voice, nominalization, and connective-deletion families, the full model must show a lower inversion rate than the augmentation-matched baselines across at least three seeds, with no loss of in-distribution directed F1 on EventStoryLine or RECESS span F1 beyond seed variance. The third check tests the semantic-label component. If learned slot vectors match definition embeddings on robustness, on paraphrased definitions, and on transfer to unseen MAVEN-ERE relation types, the label-embedding ingredient is decorative and should be dropped from the claim. The fourth check concerns over-correction: the distractor false-positive rate on non-causal CNC sentences must not rise, which ensures the role term has not simply learned to treat every event-bearing sentence as causal in both directions.