Relation extraction reads a document and lists which pairs of entities stand in which relations, for example "Marie Curie — born in — Warsaw". Good document-level extractors are "one-pass": a single transformer forward pass reads the whole document, and a small head then scores every entity pair for every relation type at once. Their weak point is training data. In DocRED, roughly 64.6% of true relations were never annotated, so a supervised model is taught that real facts are false (Revisiting DocRED). Reinforcement learning (RL) has been shown to help here. It trains the model against a reward rather than fixed labels, so it can be credited for finding relations nobody annotated. The cost has been prohibitive, though. RL for language models is slow because the model must generate its own outputs one token at a time, and memory-heavy because it keeps extra networks such as a critic (a model that predicts expected reward) and a frozen reference copy.
This proposal rests on one observation. For a one-pass extractor, a "rollout" (a complete sampled output the reward can score) needs no generation at all. One forward pass gives a probability for every pair and relation. A full candidate relation graph is then just a vector of coin flips drawn from those probabilities, and drawing sixteen such graphs costs almost nothing beyond the one pass. This removes the sequential-generation bottleneck and the multi-sample tax of group-based RL. What remains is the question of whether RL buys anything. We argue it does only when the reward cannot be broken into independent per-pair terms. So the proposal pairs free rollouts with document-level rewards: recall of the known positives, logical consistency among predicted triples, and agreement with entity-type constraints. We test this in the partial-label setting where RL has already been shown to matter.
Core Claim
A one-pass document extractor such as ATLOP can be trained with group-relative policy gradients at close to the wall-clock time and memory of ordinary supervised training. It needs no separate critic, no in-memory reference model, and no iterative relabeling cycles. With a non-decomposable document-level reward, it should match or beat the actor-critic MLPAC under severe label scarcity on Re-DocRED. The claim is falsifiable in two directions. If a differentiable per-pair surrogate of the same reward does equally well, the RL machinery is unnecessary. If the compute cost is not close to supervised training, the efficiency argument fails.
Where This Sits
Two results motivate the proposal. MLPAC casts multi-label relation classification with partial labels as a one-step decision problem. Its classifier is an actor, a separate critic network assigns each class a reward, and a recall-based global reward stops the model from predicting nothing. With only 10% of positives kept on Re-DocRED it reaches 66.0 F1, against 7.3 for supervised training that treats unlabeled pairs as negative. Its authors name the cost of iterative actor-critic cycles and dual-network retraining as a limitation, and they score pairs independently. We keep MLPAC's problem and its global reward. We drop the separate critic and the iterative cycles, and we score whole documents rather than pairs.
The one-pass architectures supply the method. ATLOP encodes the document once. It pools the attention between subject and object to build each pair's context, and it uses a learned threshold class that separates positives from negatives for each pair; it reports 63.4 F1 on DocRED. Wang et al. showed earlier that one-pass multi-relation extraction trains 3.5× faster than pairwise encoding. We use ATLOP unchanged as the policy. Its per-pair, per-relation probabilities, read relative to the threshold class, define a product of Bernoulli distributions, which is the policy we sample from.
The RL efficiency literature constrains the design. Is Reinforcement Learning (Not) for NLP finds that sequential sampling and a vocabulary-sized action space are the main sources of cost and instability in text RL. Our actions are binary decisions produced in parallel, so both problems largely disappear. That paper's evidence concerns generation over large vocabularies, so our setting falls outside it. SAPO removes the separate critic by reading values off the actor's own hidden states. BPCO makes a critic stable with one response per prompt, matching 16-sample GRPO (group relative policy optimization, which compares several sampled outputs against their group mean instead of using a critic). Both work because generation is expensive, so each saved rollout matters. Our position is the reverse. Rollouts are nearly free, so the group baseline that GRPO relies on is cheap, and the critic can be dropped entirely rather than engineered. GRPO's other known failure is zero gradient when every sample in a group gets the same sparse reward. That failure is unlikely here because the rewards are continuous, and we will measure how often it occurs.
Research Program
For each training document, we run one forward and one backward pass of the ATLOP policy. From its probability matrix we draw $G$ relation graphs, with $G$ between 1 and 32. Each graph gets a document-level reward $R$, and the advantage of a graph is its reward minus the group mean, divided by the group standard deviation. The loss is the standard policy gradient on the sampled graphs' log-probabilities plus a KL penalty toward a reference policy. Because the extractor is one-pass, the reference is the supervised warm-start model's output for each training document. That output can be computed once, cached, and read from disk, so no reference model occupies GPU memory. The objective is
$$\mathcal{L}(\theta) = -\frac{1}{G}\sum_{g=1}^{G} \hat A_g \log \pi_\theta(Y_g \mid d) + \beta\, \mathrm{KL}\big(\pi_\theta(\cdot\mid d)\,\|\,\pi_{\text{ref}}(\cdot\mid d)\big),$$
where $d$ is a document, $Y_g$ is the $g$-th sampled relation graph, $\pi_\theta$ is the Bernoulli policy produced by the extractor's single pass, $\hat A_g$ is the group-normalized reward, $\pi_{\text{ref}}$ is the cached warm-start policy, and $\beta$ sets the strength of the anchor.
The reward is where the scientific content lies. It has three parts.
The first is recall on the observed positive labels, as in MLPAC's global reward. This rewards the model for keeping known facts without penalizing it for predicting unlabeled ones.
The second is a consistency score over the whole predicted graph. It gives credit for predicted triples that follow logically from other predicted or labeled triples, and it penalizes violations. The rules are inverse relations, symmetric relations, and the kinds of logical rules Revisiting DocRED used to expand its annotations. Rules are mined only from the training split.
The third is an entity-type compatibility penalty that stops the graph from inflating with type-impossible triples.
The second and third parts depend jointly on many predicted triples at once. This non-decomposability is what makes a sampling-based estimator necessary rather than a convenience. If every reward term were a sum of per-pair terms, the policy gradient under a factorized policy would reduce to the analytic gradient of expected reward. In that case we should just optimize that expectation directly, and the ablation below tests exactly this.
Two further analyses are built in. The first measures how far the model can drift from its anchor before it over-predicts; this is the "prior" that positive-unlabeled methods need, which we instead expose as $\beta$. We sweep $\beta$ and report predicted triples per document against Re-DocRED's 34.7 per document. The second is a cost profile of wall-clock time per epoch and peak memory, set against supervised ATLOP and against a reimplemented MLPAC on the same encoder and hardware.
Baselines
The naive reference point is ATLOP trained with supervision on the partial labels, with every unlabeled pair treated as negative. This is MLPAC's "negative mode", which MLPAC reports at 7.3 F1 on Re-DocRED with 10% of positives. We rerun it on our encoder so that the gap is measured in our setup rather than borrowed. As an upper reference we train the same ATLOP on the full Re-DocRED labels. Revisiting DocRED reports that such models reach the mid-70s F1. This tells us how much of the fully supervised ceiling each method recovers.
The strongest existing competitor is MLPAC, which reports 66.0 F1 at 10% positives on Re-DocRED. It is the only method in the file that attacks this exact problem with RL and wins by a large margin. For fairness we run MLPAC with the same pretrained encoder, the same data splits and label-dropping seeds, and a matched budget of tuning trials. Beating it means higher F1 on the fully annotated Re-DocRED test set at 10% and at the other label ratios, at no more than half its wall-clock training time and peak memory. Matching its F1 at a fraction of the cost would count as a partial win on efficiency alone.
Positive-unlabeled learning with SSR-PU is run with its prior estimated from the partially labeled training data, as a practitioner would have to do, and also with the oracle prior from Re-DocRED to show its best case. No reported result found on Lacuna for SSR-PU on Re-DocRED.
Self-training uses ATLOP's own threshold class to choose pseudo-labels, with a matched number of rounds and matched compute.
Two ablations isolate the key ingredient. The first replaces the sampled policy gradient with direct gradient descent on a differentiable surrogate of the same reward: expected recall, plus consistency terms relaxed into products of probabilities. It has the same data, compute, and anchor. The second drops the consistency and type terms and keeps only the recall reward. We also vary $G$ over 1, 4, 16, and 32 to check the claim that larger groups are nearly free and whether they help.
Why not just…?
Why not just use positive-unlabeled learning? Methods like SSR-PU need the true fraction of positives, and MLPAC reports that it beats prior-dependent positive-unlabeled baselines because that fraction is unknown in practice. Our method still has a knob, $\beta$, that controls how far predictions spread. The honest comparison is therefore SSR-PU with an oracle prior against our method with $\beta$ chosen on the partially labeled dev set only. If oracle-prior SSR-PU wins by a wide margin, the gain from RL is a tuning convenience rather than a modeling advance, and we will report it that way.
Why not just self-train with confident pseudo-labels? Self-training is the cheapest competitor and a real threat, because ATLOP's learned threshold already gives a sensible confidence cutoff. The evidence against it in the file comes from dialogue generation. There, self-training (3.73 BLEU-4) barely moved over its base model (3.69), while reward-driven training reached 4.30 (Teacher Forcing Recovers Reward Functions). That setting is not relation extraction. Self-training also cannot use the consistency reward, because a pseudo-label is accepted or rejected pair by pair. The matched self-training baseline settles the question directly.
Why not just optimize a differentiable expected reward and skip RL? This is the sharpest objection. For per-pair rewards, it is mathematically equivalent to our method with lower variance. We accept that. Our claim depends only on the non-decomposable consistency and type terms. The surrogate ablation is the decisive test. If the relaxed surrogate matches the sampled policy gradient, the conclusion is that one-pass extractors do not need RL at all, which is itself a useful negative result. Our proposal stands only if sampling wins specifically when the consistency terms are switched on.
Why not just shrink MLPAC's critic onto the shared encoder, SAPO-style? That would recover part of the memory saving. It would keep a learned critic, though, which BPCO shows needs careful design to stay stable. It would also keep MLPAC's iterative label-enhancement cycles. When rollouts cost nothing, a group-mean baseline is unbiased and needs no training. We will include a variant of our method with a light value head in place of the group baseline, to show whether a learned critic adds anything once rollouts are free.
Proposed Experiments
The main benchmark is Re-DocRED. We keep 10%, 30%, and 50% of positive training labels over three random seeds, and evaluate on the fully annotated test set with F1 and with ignoring-train-overlap F1. The second setting has naturally occurring missing labels: we train on original DocRED annotations and test on Re-DocRED, where roughly 64.6% of true triples were missing from training. As a transfer check, we train on CDR and GDA with dropped labels. ATLOP reports gains of 4.3 and 1.4 F1 over prior work there, but no partial-label results are reported on Lacuna for either dataset.
Every run reports F1, predicted triples per document, the rate of consistency violations on the test set, wall-clock time per epoch, and peak GPU memory. All are measured on identical hardware against supervised ATLOP and against MLPAC.
Three checks are decisive. First, cost: our method at $G=16$ must train within 1.5× the time and memory of supervised ATLOP; otherwise the "free rollouts" premise is false. Second, value over a surrogate: with consistency rewards on, sampled RL must beat the differentiable surrogate by a margin larger than the variation across seeds, and the gap should vanish when only the recall reward is used. This is the prediction that ties the benefit specifically to non-decomposable rewards. Third, value over the state of the art: F1 at 10% labels must meet or exceed MLPAC's reported 66.0 at a fraction of its cost. If the second check fails, we will publish the simpler conclusion: one-pass extractors under partial labels can drop RL for direct expected-reward optimization.
References
- Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction
- Combining Supervised Learning and Reinforcement Learning for Multi-Label Classification Tasks with Partial Labels
- Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling
- Extracting Multiple-Relations in One-Pass with Pre-Trained Transformers
- Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
- SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning
- Best Practice Critic Optimization
- Teacher Forcing Recovers Reward Functions for Text Generation