Document-level relation extraction reads a whole document and labels every pair of entities with the relations between them, such as "A is the spouse of B." Most extractors decide each pair separately. Prior work treats this "relational independence" as the main weakness of these systems, because facts in a document depend on each other: they are symmetric, they have inverses, and they chain through an intermediate entity. Ensembles of differently designed extractors fix many of these errors, but they multiply inference cost by the number of members. The usual cure is knowledge distillation: train one student network to copy the ensemble's averaged probabilities. This proposal argues that the cure brings the disease back. Averaging the members into one probability per pair keeps each pair's marginal and throws away which pairs the members got right or wrong together. That joint agreement is where an ensemble encodes its relational consistency. A student trained on per-pair averages has learned the independence assumption from its teacher.
We propose to distill the ensemble's joint structure. The student runs one encoder pass and adds a small low-rank latent head, so its predictions on pairs that share an entity can co-vary the way the ensemble members co-vary. At test time we draw many samples from this tiny head, which costs almost nothing next to the encoder, and decode the most coherent relation set. The claim is specific: on benchmarks with complete annotations, joint distillation should keep ensemble-level consistency and multi-hop accuracy at single-model cost. Ordinary marginal distillation should match ensemble F1 on easy pairs but lose on consistency and inference-heavy pairs.
Why the obvious idea is not enough
The intersection already has distillation work, but none of it targets joint structure. The closest is Distilling Knowledge for Search-based Structured Prediction. It compresses an ensemble of parsers or translators into one greedy student by matching soft targets on gold and explored states. Our method borrows its premise that soft ensemble targets carry structure hard labels lose. That work covers only left-to-right transition sequences, where each soft target is conditioned on the history, so joint structure comes free. A grid of entity pairs has no history to condition on. The per-pair targets are genuinely marginal, and that gap is ours.
KD-Roberta uses a clean-data teacher to denoise distant supervision, with axial attention over the pair grid. Self-distillation relational reasoning distills from a masked branch of the same network, using four hand-fixed reasoning patterns that cover about 76.5% of DocRED triples. Neither compresses an ensemble, and both still distill per-pair distributions.
On the relational side, three lines break independence at test time at a cost. MILR mines logical rules and runs an integer linear program at inference. Iterative Inference refines the pair matrix over three cross-attention passes. DocuNet, GREP and DORE build pair grids, pair graphs or ordered generation. These motivate the ensemble: their inductive biases differ, so their errors should be partly decorrelated.
Revisiting DocRED is both motivation and constraint. It used an ensemble of extractors to surface the roughly 64.6% of triples missing from DocRED, which shows the diversity is real. It also showed that on the original labels, ensembles are penalized for finding true but unannotated facts. Any honest study of joint ensemble structure must therefore run on Re-DocRED.
Core Claim
Let $M$ ensemble members each output a probability $p^{(m)}_{ijr}$ that relation $r$ holds from entity $i$ to entity $j$. Marginal distillation trains the student on $\bar p_{ijr}=\frac1M\sum_m p^{(m)}_{ijr}$. Our student instead outputs a mean logit $\mu_{ijr}$ and a $k$-dimensional loading vector $u_{ijr}$, with $k$ around 8 to 16, from one encoder pass. It defines
$$\ell_{ijr}(z)=\mu_{ijr}+u_{ijr}^\top z,\qquad z\sim\mathcal N(0,I_k),$$
where $z$ is a shared document-level latent. Two pairs whose loadings align will tend to switch on and off together, as symmetric or chained facts should.
Each member's thresholded relation matrix $Y^{(m)}$ is treated as one sample from the teacher's joint distribution. The student is trained to assign each such matrix high likelihood under its own mixture over $z$. The likelihood is estimated with $S$ Monte Carlo draws:
$$\mathcal L=-\frac1M\sum_{m}\log\frac1S\sum_{s=1}^{S}\prod_{i,j,r}\mathrm{Bern}\!\left(Y^{(m)}_{ijr}\,;\,\sigma(\ell_{ijr}(z_s))\right),$$
where $\sigma$ is the logistic function. A standard per-pair loss toward $\bar p$ is added so the marginals stay calibrated.
The product over pairs inside the average over samples is what lets the student represent "these two pairs flip together." A marginal loss cannot express this. Pairs with no relation (about 97% per Revisiting DocRED) are scored only where at least one member gives them non-trivial probability. This keeps the cost near linear in candidate pairs rather than quadratic.
At inference we draw $S'$ latent samples, around 64, through the head only. We threshold each sample and return the sampled configuration with the highest joint score, which plays the role of an ensemble vote. This gives the solver-free, consistency-aware decoding that MILR obtains from an integer program.
Research Program
The teacher is a heterogeneous ensemble of four extractors trained on Re-DocRED with different seeds and architectures: an ATLOP-style pair classifier, DocuNet, an axial-attention extractor in the style of KD-Roberta, and Iterative Inference. All four share one RoBERTa-large encoder size, so differences come from relational design.
The first step is diagnostic and could end the project early. We will measure how much of the ensemble's advantage over its best member sits in joint structure. For each member, we compute the correlation of errors across pairs that share an entity, and how often each member violates MILR-mined rules and the symmetry and inverse constraints used by CoIn. We then compare against what a model with independent errors and the same marginals would produce, by sampling every pair independently from $\bar p$.
If the ensemble vote is no more consistent than that independent model, joint structure carries nothing worth distilling. We would then report a negative result rather than proceed. If it is clearly more consistent, we train the joint student and the ablations below.
Second, we study what the latent loadings learn. We test whether pairs with aligned loadings line up with mined rules, such as symmetric pairs, inverse pairs, and composition chains through a shared entity. We also test whether the student discovers co-variation outside the four hand-coded patterns of Zhang et al., which would address that paper's 76.5% coverage ceiling.
Third, we test transfer. We take the same recipe to DWIE, whose relations span hundreds of tokens, and to DialogRE, where symmetry and inverse constraints are dense. Both settings check whether the gains come from relational structure rather than Wikipedia-specific patterns.
Baselines
The naive reference point is a single ATLOP-style RoBERTa-large extractor trained directly on Re-DocRED, with no ensemble and no distillation. It sets the floor for accuracy and the cost of one forward pass.
The ceiling is the full four-member ensemble run at test time with probability averaging and thresholding. A second ceiling adds MILR's integer-program decoding on top. The joint student should sit between these points on accuracy while costing the same as the naive model.
The strongest existing single-model competitors are the published relational extractors run under our pipeline. We have reported numbers for them on two benchmarks. On original DocRED, the file reports KD-Roberta at 67.28 F1 on the DocRED leaderboard, and Iterative Inference at 48.75 Infer-F1. On Re-DocRED, the file reports GREP at 81.61 F1. We found no reported result on Lacuna for an ensemble of these extractors scored on Re-DocRED, so we will establish that number ourselves.
The key ablation is marginal ensemble distillation. It uses the same student encoder, the same ensemble, the same unlabeled distantly supervised documents, and the same steps and tuning budget, with the loss computed only on $\bar p$. It isolates whether the joint term matters. A second ablation keeps the latent head but trains it with marginal loss only, to separate the extra parameters from the joint objective. A third replaces our decoding with plain thresholding of $\mu$.
All students get one hyperparameter search of equal size on the Re-DocRED dev set and run on identical hardware. Latency is reported per document, including the sampling step.
"Beating" has three conditions:
- The joint student exceeds marginal distillation on Infer-F1 and rule consistency.
- It does so without losing more than a fraction of a point of overall F1.
- It closes most of the gap to the test-time ensemble at near single-model latency.
Why not just…?
Why not just distill the averaged ensemble probabilities, which is standard and known to work? This is the real competitor and the main ablation. The evidence from Liu et al. that soft targets help is exactly why we keep the marginal term. What we question is whether averaging per pair loses the co-variation across pairs. If marginal distillation already matches the ensemble on consistency and Infer-F1, the proposal fails, and the first diagnostic step is built to show that early.
Why not just add rule-consistency losses like MILR or CoIn to a single model? These add no inference cost and are strong. MILR reports 1.1 to 3.8 F1 gains across backbones. However, its rules are mined under a closed-world assumption that breaks on sparse data, and MILR still relied on a test-time integer program for its best consistency. The ensemble provides consistency signal learned from data, including patterns no rule miner proposed. We will run marginal distillation plus MILR's regularizer as an extra baseline. If it equals the joint student, then hand-mined rules suffice and our latent head adds nothing.
Why not just keep Iterative Inference, whose extra passes only touch a light head? This is a fair threat, since its three refinement passes run over the pair matrix rather than the encoder. We report it as a member and as a standalone baseline, with measured latency. The claim against it is not speed alone. It is one architecture, while the ensemble pools four biases. The comparison that settles this is the joint student versus Iterative Inference on Re-DocRED Infer-F1 at matched latency.
Why not just find the single best member, as in searching for a strong expert? Cendra et al. show a well-chosen single expert can rival an ensemble. Our best-member baseline covers this directly. If one member matches the ensemble, there is nothing to distill and the diagnostic will show it.
Proposed Experiments
Training uses Re-DocRED train together with ensemble-labeled distantly supervised DocRED documents. Labeling those documents with the ensemble also partly addresses the false negatives in the original annotations. We evaluate on Re-DocRED dev and test, original DocRED (for comparability with reported numbers), DWIE, and DialogRE.
The metrics are:
- F1 and Ign F1;
- Infer-F1 on facts that need multi-sentence reasoning;
- rule-violation rate under MILR-mined rules, plus CoIn's symmetry and inverse checks;
- held-out log-likelihood of member matrices, to test whether joint structure is captured;
- latency and peak memory per document.
There are two decisive checks. The first is the diagnostic: does the ensemble's consistency exceed the independent-error model with the same marginals? The second is the gap between joint and marginal distillation on consistency and Infer-F1 under an equal budget, averaged over five seeds with bootstrap confidence intervals.
A stress test reruns everything on original DocRED. If ensemble gains look smaller there and larger on Re-DocRED, that supports the view from Revisiting DocRED that incomplete labels hide relational gains. Finally, we vary the number of test-time samples $S'$ from 1 to 256. This traces how much of the ensemble's consistency we recover per unit of extra cost in the head.
References
- Distilling Knowledge for Search-based Structured Prediction
- Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillation
- Exploring Self-Distillation Based Relational Reasoning Training for Document-Level Relation Extraction
- Boosting Document-Level Relation Extraction by Mining and Injecting Logical Rules
- Towards Better Document-level Relation Extraction via Iterative Inference
- Document-level Relation Extraction as Semantic Segmentation
- Document-Level Relation Extraction with Global Relations and Entity Pair Reasoning
- DORE: Document Ordered Relation Extraction based on Generative Framework
- Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction
- Consistent Inference for Dialogue Relation Extraction
- DWIE: An entity-centric dataset for multi-task document-level information extraction
- Can Bayesian Optimization Efficiently Find a Strong Single Expert in Neural Thickets?