Document-level relation extraction asks a model to read a whole document and decide how every pair of named entities is related, for example that a chemical causes a disease. Often the answer depends on evidence spread across several sentences. When such a model predicts a relation, a user wants to know which sentences the prediction actually depends on: which ones, if deleted, would change the answer. The honest way to find out is to delete each sentence, or each set of sentences, from the text and rerun the model. That costs one full transformer pass per deletion, and searching over sets of sentences grows exponentially. What current models offer instead are attention scores, supervised evidence predictions, or heuristic paths. None of these is shown to track what happens when the text is actually changed.
We propose to build a relation extraction model around a small graph of entity and sentence nodes, a heterogeneous document graph (a graph whose nodes have different types). The graph is trained so that masking a sentence node reproduces the effect of deleting that sentence from the text. Once trained this way, the graph answers counterfactual questions ("would this relation survive without sentences 3 and 5?") for every entity pair and every sentence set from a single encoder pass plus a few operations on a graph of roughly thirty nodes. The speed gain applies to explanation, not training. The interpretability claim is empirical and checkable against real text deletions.
Core Claim
A document graph whose propagation is linear over typed edges, and which is trained with a consistency loss tying graph masking to text deletion, can match strong transformer-only extractors on DocRED, CDR and GDA. Its graph-predicted counterfactuals should agree closely with true re-encoding counterfactuals, and it should produce relation-specific, set-level explanations one to two orders of magnitude faster than occlusion or per-instance mask optimization.
What is new is the training target: the graph must be a faithful proxy for text edits. Prior document graphs such as EoG, which refines edge representations along multi-hop paths, and DHG, which uses separate structure and reasoning layers with typed edge weights, use the graph only to improve accuracy. Nobody checks whether their structure explains anything.
Why This Gap Is Real
Three lines of evidence motivate the proposal. Relation Extraction with Explanation showed that unconstrained attention in relation extractors correlates poorly with human rationales (Kendall tau 0.08, raised to 0.31 only with distractor training). Probing GNN Explainers derived faithfulness bounds for graph explainers and found that a random-edge baseline often beats published explainers. SGR extracts heuristic entity-to-entity meta-paths, reaches 57.15 F1 on DocRED, and covers about 75.2% of annotated evidence. Its paths are fixed rules, though, and it never tests whether they are necessary for the prediction.
Three other lines constrain the design. ATLOP shows that a graph-free transformer with context pooling reaches 63.4 F1 on DocRED, well above graph models like SGR, so our graph cannot come at an accuracy cost. DREEAM supervises ATLOP's context attention with evidence labels. It improves F1 by about a point, but its evidence is relation-agnostic and supervised rather than counterfactual. Finally, per-instance optimizers such as CF-GNNExplainer, which learns a perturbation mask to find the fewest edge deletions that flip a prediction, must be rerun for every example. That is exactly the cost we want to remove.
Research Program
Model. A pre-trained encoder (RoBERTa, or BERT for comparability with DREEAM) reads the document once. We pool mention tokens into entity nodes, as in ATLOP, and sentence tokens into sentence nodes. Edges have three types: entity-in-sentence, sentence adjacency, and entity co-occurrence within a sentence. Reasoning happens through a linear, SGC-style propagation (simplified graph convolution: repeated neighbor averaging with no nonlinearity between steps):
$$Z = \sum_{k=0}^{K} \alpha_k \Big(\sum_{r} \beta_r \hat{A}_r\Big)^k X,$$
Here $X$ holds the encoder node embeddings, $\hat{A}_r$ is the normalized adjacency for edge type $r$, $\beta_r$ are learned type weights (as in DHG's weighted relational convolution), $\alpha_k$ are learned hop weights, and $K$ is the number of propagation steps. A bilinear classifier with ATLOP's adaptive threshold scores each pair from its entity rows of $Z$ plus a localized context vector.
Because the propagation is linear, masking any set of sentence nodes changes $Z$ in a way computable exactly and in closed form: it removes every walk through the masked nodes. With about 30 nodes, masking all sentences one at a time, or searching greedily over sets, is negligible next to one encoder pass. Nonlinear power stays in the encoder and classifier. This mirrors the linear-propagation setting in which Graph Coarsening with Message-Passing Guarantees proves its bounds. We read the sentence graph as a coarsening of the token graph and test that operator's asymmetric pooling $QSQ^+$ (where $Q$ maps tokens to super-nodes and $Q^+$ is its pseudo-inverse) as an ablation for the token-to-node step.
The leakage problem and the consistency loss. Masking a sentence node is not the same as deleting the sentence. Through self-attention, every other node's embedding still carries information from the masked sentence. We therefore train with a counterfactual consistency term. At each step we sample a document $D$ and a sentence set $s$ (size 1 to 3) and run the encoder on the edited text $D_{\setminus s}$. We then penalize
$$\mathcal{L}_{cf} = \mathrm{KL}\big(\,\mathrm{sg}[f(D_{\setminus s})]\;\|\;g(\mathrm{mask}_s(D))\,\big),$$
where $f$ is the full model applied to the edited text, $g(\mathrm{mask}_s(D))$ is the model's prediction on the original encoding with nodes in $s$ masked, and $\mathrm{sg}$ stops gradients into the target. Both sides are computed for all entity pairs at once. This roughly doubles training cost. The loss pushes the model to route cross-sentence evidence through explicit graph edges rather than hidden encoder attention. The amount of residual leakage is itself a measured quantity.
Explanations. For each predicted relation, meaning each pair and each relation label, we report two things computed from the graph alone. The first is the minimal necessary sentence set, found by greedy search over masked sets until the label flips. The second is the minimal sufficient set, found by keeping only chosen sentences. Both are relation-specific, which addresses DREEAM's relation-agnostic evidence.
Baselines
Naive reference points. The first is a random sentence set of matched size, following the random-edge check in Probing GNN Explainers. The second is "all sentences mentioning either entity", a zero-cost heuristic. An explainer must clearly beat both on deletion faithfulness, the drop in the predicted relation's probability when the selected sentences are removed from the text.
Strongest extraction baselines. These are ATLOP, which reports 63.4 F1 on DocRED, and DREEAM, which reports +0.6/+1.0 Ign F1/F1 over the prior best with BERT-base. No reported absolute DREEAM F1 was found on Lacuna, so we rerun it. Both are the strongest competitors because they show a graph is unnecessary for accuracy. We run them and our model with the same encoder, data, learning-rate grid, and number of seeds. On CDR and GDA the graph reference points are DHG, which reports 65.9 F1 on CDR and 83.1 F1 on GDA, and EoG, which reports 63.6 F1 on CDR. Beating the extraction baselines means staying within 0.5 F1 of the rerun DREEAM on DocRED while winning clearly on explanation metrics. We do not claim an accuracy gain.
Strongest explanation baseline. This is occlusion on DREEAM: delete sentences from the text and re-encode. It is both the ground truth for counterfactual agreement and the speed competitor. We give it greedy set search under the same budget of flipped labels. Two further explanation baselines are DREEAM's evidence attention scores and SGR's heuristic paths, which reach 75.2% evidence coverage in that paper. We also run CF-GNNExplainer on our own graph without the consistency loss, to separate "a graph explainer" from "a graph trained to be faithful." Beating these means higher deletion faithfulness than DREEAM attention and SGR paths, sign and flip agreement with occlusion approaching occlusion's own agreement across seeds, and at least 10× lower explanation wall-clock time.
Key ablations. The first removes $\mathcal{L}_{cf}$. This isolates the proposal's central ingredient, and we expect a large drop in agreement with occlusion at similar F1. The second replaces linear propagation with a two-layer nonlinear graph network, which loses exact masking. The third uses sentence-block-restricted encoder attention, which removes leakage by construction but should cost accuracy.
Why not just…?
Why not just occlude sentences in the transformer and rerun it? This is the most serious alternative. On DocRED and CDR, documents are short, and single-sentence occlusion costs only one extra encoder pass per sentence per document, covering all pairs at once. We treat it as ground truth, not as a straw man. Our advantage appears only for set-level explanations, whose search grows combinatorially, and for explaining many documents at deployment time. We will report explanation time as a function of the number of sentences and the size of the explanation set. If agreement with occlusion is high but the speed gap is small at DocRED scale, we will say so.
Why not just supervise attention with evidence, as DREEAM does? DREEAM's evidence is relation-agnostic and learned to imitate annotators. It is never tested against what the model does when text is removed. We measure how well DREEAM's attention agrees with occlusion. The Kendall tau of 0.08 in Relation Extraction with Explanation warns that attention can be far from rationale. If DREEAM's attention already agrees with occlusion as well as our graph does, the consistency loss adds little, and the experiment will show it.
Why not just use SGR-style heuristic paths? They are fixed rules, they miss about a quarter of annotated evidence, and they were not tested for necessity. We score SGR paths on the same deletion-faithfulness and occlusion-agreement metrics.
Why not just train the transformer with sentence dropout and skip the graph? That makes the model robust to deletions, but it still requires re-encoding to answer each counterfactual. The graph's role is to make every counterfactual cheap to compute. The no-graph variant appears in our speed comparison.
Proposed Experiments
We train on DocRED (human-annotated split), CDR and GDA, and report extraction F1 (Ign F1 on DocRED). For counterfactual agreement, we sample 2,000 pair–sentence-set deletions on held-out documents. For each we compute both the graph-predicted and the true re-encoded change in relation probability, and report Spearman correlation, sign agreement, and label-flip agreement. The ceiling is occlusion agreement between two seeds of the same model. For faithfulness we report deletion and insertion curves against the random and heuristic baselines. For plausibility we report evidence F1 against DocRED's supporting sentences, split by whether a pair carries multiple relation labels, which tests relation specificity. We also report wall-clock time per document for all explanations.
The decisive check is the ablation on $\mathcal{L}_{cf}$. If the unregularized graph already agrees with occlusion, leakage is small and the contribution reduces to a fast explainer. If agreement collapses without the loss, and F1 stays within half a point of DREEAM with the loss, the claim holds: a document graph can be trained to be a faithful, cheap stand-in for editing the text.
References
- Connecting the Dots: Document-level Neural Relation Extraction with Edge-oriented Graphs
- Document-level Relation Extraction with Dual-tier Heterogeneous Graph
- Relation Extraction with Explanation
- Probing GNN Explainers: A Rigorous Theoretical and Empirical Analysis of GNN Explanation Methods
- Document-level Relation Extraction via Subgraph Reasoning
- Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling
- DREEAM: Guiding Attention with Evidence for Improving Document-Level Relation Extraction
- CF-GNNExplainer: Counterfactual Explanations for Graph Neural Networks
- Graph Coarsening with Message-Passing Guarantees