Screening Off Spurious Event Causality with Language-Model Independence Tests

Systems that read a news article and draw a graph of which events caused which still get most causal links wrong. On the MAVEN-ERE benchmark, a RoBERTa baseline reaches only about 31.5 causal F1. A recurring error is to link two events simply because they happen near each other and share a background cause. Causal discovery, the statistical field that recovers cause-and-effect graphs from data, has a standard tool for exactly this error: the conditional independence test. If knowing a third event C makes A irrelevant to B, then the A–B link is explained by C and should be deleted. This is called "screening off." Event extraction from text has never used this test. The closest attempt, Dr.ECI, asks an LLM agent to reason qualitatively about chains, forks and colliders. No statistic is computed and no link is pruned by a test.

We propose running a real PC-style discovery procedure inside each document. PC is the classic algorithm that starts from a dense graph and removes an edge whenever some conditioning set makes its endpoints independent. The independence test is computed from a language model's probabilities under controlled edits of the text. A temporal relation extractor then supplies a soft "arrow of time" that orients the edges that survive. The core claim is that conditioning, not just pairwise judgment, removes common-cause false positives. We hypothesize that it does so without costing recall on direct links, and that this gain survives against strong LLM and constrained-learning baselines.

Core Claim

Treat a pretrained language model as a density estimator over "what else is likely to have happened, given this account." Then a conditional mutual information between two event mentions, given a third, can be estimated by scoring edited versions of the document. Concretely, let $x$ be the document, $A$, $B$ and $M$ event mentions, and $x_{\setminus A}$ the document with $A$'s clause replaced by a neutral paraphrase that asserts nothing about $A$. We define the screening score

$$ S(A,B \mid M) = \left| \log p_\theta(B \mid x_{\setminus B}) - \log p_\theta(B \mid x_{\setminus A,\setminus B}) \right|, $$

Here $p_\theta(B \mid \cdot)$ is the language model's probability of $B$'s trigger clause when that clause is removed and asked for back. $M$ is kept verbatim in both contexts. When $S$ is small for some $M$, $A$ carries no information about $B$ beyond what $M$ already carries. The candidate edge A–B is then deleted, and $M$ is recorded as its separating set.

This is the text analogue of the conditional-mutual-information tests that TRACE (Your Autoregressive Model Already Reveals the Causal Graph) runs over autoregressive model outputs on discrete event logs. TRACE supplies the method we borrow. It works on a single stream of numeric event codes, assumes no same-time effects, and never touches documents. We move the test into prose. We also use the language model's world knowledge as the "population" the test is computed over, instead of repeated observations of the same system.

Research Program

The pipeline has three stages.

The first stage is high-recall candidate generation. A causal classifier, trained with the joint temporal-causal-subevent constraint losses of Joint Constrained Learning for Event-Event Relation Extraction, proposes every pair above a deliberately low threshold. This gives the "fully connected" starting graph, restricted to plausible pairs so the test budget stays bounded.

The second stage is skeleton pruning. We follow the PC schedule implemented in Causal-learn: order-zero tests first, then conditioning on one, then two, of the neighboring candidate events. Conditioning sets are capped at two and drawn only from current neighbors. On the Number of Conditional Independence Tests in Constraint-based Causal Discovery motivates this cap, since it studies how test budgets can be limited without losing consistency. Each test costs a few scoring passes, not generation, so a document with fifty candidate edges needs hundreds rather than thousands of forward passes. The deletion threshold is calibrated on development data. We will also report a version that uses the raw score softly, like the continuous relaxations in Differentiable Constraint-Based Causal Discovery. That paper shows that hard discrete test decisions compound errors when samples are few. Our setting has the same brittleness, because one bad edit can delete a true edge.

The third stage is orientation. Surviving edges are oriented in three ways. Collider rules come from the recorded separating sets. Tiered background knowledge is taken from predicted temporal order, following the tiered-knowledge results of Andrews et al. and Bang et al.. Meek-style propagation fills in the rest. Temporal tiers here are predicted, not given, so we weight each backward-edge prohibition by the temporal classifier's confidence. We exempt events tagged as intended or future, the "future projection" failure that Joint Reasoning for Temporal and Causal Relations identified in its hard precedence constraint. Same-sentence pairs with no confident temporal order are oriented only by colliders or left undirected. This is the honest answer to the instantaneous-effect limitation that TRACE states outright.

Two analyses accompany the system. The first is a validity study of the edit-based test on synthetic narratives generated from known causal graphs, in the style of the TinyStories-derived COPES-hard data. Because the forks, chains and colliders are known exactly, we can measure the test's false-deletion and false-retention rates as a function of language model size. The second analysis concerns what the gold labels mean. MAVEN-ERE and EventStoryLine may label indirect as well as direct causal pairs. Deleting a chain-mediated A–B edge is correct under a "direct cause" reading and wrong under an "any cause" reading. We therefore separate chain deletions from fork deletions and report each against both readings.

Baselines

The naive reference point is the first-stage candidate generator itself: the constraint-trained RoBERTa classifier at its F1-optimal threshold, with no screening. It isolates whether the pruning stage adds anything. The MAVEN-ERE paper reports roughly 31.5 causal F1 for its RoBERTa-base baseline (MAVEN-ERE). We will retrain it with the same encoder, data and seeds as our pipeline.

The strongest existing text methods come in two kinds. The first is Dr.ECI (Dr.ECI), which reports 35.6 causal F1 on MAVEN-ERE and 84.4 intra-sentence F1 on EventStoryLine. It is the most direct competitor because it targets the same chain/fork/collider errors, only through prompting. The second is TacoERE (TacoERE), whose GPT-4 variant reaches 41.9 F1 on MAVEN-ERE, the highest number in our file. The file does not state which relation subset that F1 covers. We will rerun it on the causal subset before treating 41.9 as the causal bar.

For fairness, every LLM-based method uses the same underlying language model as our tests. The only exception is the published numbers, which we quote separately. Every method gets equal development-set tuning trials, and we report language model calls per document so Dr.ECI's multi-agent cost and ours can be compared directly. On COPES-hard, the reference is the synthetic-control method of Event Causality Identification with Synthetic Control, at precision 0.2663 and F1 0.3930. It is the closest counterfactual-edit method, but it tests pairs without conditioning.

Three ablations isolate the key ingredient. The order-zero ablation stops PC after unconditional tests, making it essentially a pairwise counterfactual test. The random-conditioning ablation conditions on random non-neighbor events. A third variant replaces screening entirely with transitive reduction of the predicted graph. Beating the baselines means two things. First, a statistically reliable causal F1 gain over the naive classifier and over Dr.ECI at equal model and budget. Second, most of that gain must come from precision on pairs that share a gold common cause, and it must vanish in the order-zero and random-conditioning ablations.

Why not just…?

Why not just prompt a frontier LLM with chain-of-thought? The file's evidence says direct prompting over-predicts causality. Raw GPT-4 reaches 32.8 F1 on MAVEN-ERE in the TacoERE comparison. The synthetic-control paper documents "causal parroting," where the model confuses correlation with causation, on COPES-hard. Dr.ECI reports causal hallucinations under vanilla chain-of-thought. We still include prompting with the same model as a baseline. If it matches our pipeline on fork-sharing pairs, the conditioning claim is refuted.

Why not just use Dr.ECI's structured reasoning, or transitive reduction, to drop mediated links? This is the most serious threat, because it targets the same motifs at lower engineering cost. Transitive reduction can remove chain-mediated edges. It cannot detect forks, where A and B share a cause C but neither causes the other: the graph has C→A and C→B, and nothing about the A–B edge looks redundant. Dr.ECI's reasoner can in principle notice forks, but only by the LLM's say-so. The decisive comparison is on gold fork pairs from the synthetic narratives and MAVEN-ERE. If Dr.ECI's precision there equals ours, the statistical test adds nothing, and we would reframe the work as a cheaper, deterministic reimplementation.

Why not just add ILP or t-norm temporal constraints to a pairwise classifier? Joint constrained learning already captures the precedence rule. Joint Constrained Learning for Event-Event Relation Extraction found that post-hoc ILP added negligible gains on top of trained constraints, so we build on that model rather than competing with it. Precedence can veto a backward edge, but a spurious fork link usually respects time, since both effects follow their common cause. Temporal constraints therefore cannot remove it. The naive baseline is exactly this constrained model. Any gain over it measures what screening adds beyond temporal logic.

Proposed Experiments

The decisive experiment has four parts, in order of risk.

We begin with the test's validity. On synthetic SCM-generated narratives, it must separate true independences from dependences, with false-deletion rates reported per motif type. If it cannot do so even with ground-truth structure, the program stops.

Next comes precision on common-cause pairs. On MAVEN-ERE, EventStoryLine and Causal-TimeBank, we split test pairs by whether they share a gold common cause. The screening stage must raise precision on that slice relative to the naive classifier. Recall on gold direct pairs should drop by less than that precision gain.

Third, we measure overall causal F1 against Dr.ECI and TacoERE on the causal subset, under matched language model and call budgets. Separate reporting is required for the "direct" and "any cause" label readings.

Finally, we evaluate orientation. On the dense temporal-causal annotations of Ning et al.'s augmented EventCausality data, confidence-weighted temporal tiers with the teleological exemption must orient edges more accurately than hard tiers. We report the future-projection cases separately, since that is where hard precedence is known to fail.

A negative result on the third part with a positive result on the second would still be informative. It would tell the field that benchmark labels reward indirect links that screening correctly removes.