Hospitals, law firms and companies each hold private text full of facts, such as which drug caused which reaction or which supplier serves which plant. They would like a shared model that pulls those facts out, but they cannot pool the documents. Two lines of work address halves of this problem. Generative relation extraction trains a text-to-text model to read a passage and write out its facts as a list of (head, relation, tail) triples. Federated learning trains one model across sites that never share raw data. Every federated relation extractor in the literature, however, is a classifier over a fixed label set. Every generative extractor is trained on centralized data.
This note proposes a way for sites to collaborate that exchanges neither weights nor token probabilities. Each site runs its own generative extractor on a shared public text pool and sends back only the facts it found, with a confidence for each fact. The server merges these by voting on individual facts and trains a central model on the merged fact sets with a loss that ignores the order of facts. The central claim is that, for generative extractors, the right unit of agreement between sites is the fact rather than the token. Methods that average next-token distributions across sites mix up facts that different sites happen to write in different orders, and voting at the level of whole facts avoids this. The same choice brings three practical benefits at no extra cost. The messages sent are tiny. Sites can use different models and even different relation vocabularies. No weights or gradients leave a site, and weights and gradients are what known text-reconstruction attacks need.
Core Claim
Ensemble distillation lets sites train one model by having each site's local model act as a "teacher" on shared unlabeled text, so that a central "student" learns to imitate their combined predictions. FedED uses this for medical relation classification: sites send probability vectors over a fixed set of relation classes on a public proxy set, and a central BERT student learns from their average. On the 2010 i2b2/VA clinical notes, split across three medical centers, FedED reports 75.09 F1 against 72.55 for weight averaging (FedAvg), with upload cost cut from 423MB to 42KB. The paper names joint entity-and-relation extraction as unresolved. It is unresolved for a structural reason. A generative extractor has no fixed class vector to average, and its output is a variable-length sequence.
The obvious fix is to average teachers' next-token distributions along a reference output. That breaks for the reason that Set Learning for Generative Information Extraction identifies centrally. Extracted facts form a set, but autoregressive training imposes an arbitrary order, and models learn order heuristics of their own. Suppose teacher A writes fact 1 first and teacher B writes fact 2 first. Their averaged distribution at the first position then splits mass between two different head entities. A student imitating that average can produce hybrid triples that no teacher asserted. Our claim is that aggregating at the level of whole facts removes this failure mode. We define each teacher's per-fact confidence in a way that does not depend on order. We then train the student with an order-invariant set loss. This protocol should beat token-level distillation, which the experiments test directly, and should match or exceed weight-averaging methods under non-identically distributed sites at a fraction of their communication.
Research Program
The base extractor is a sequence-to-sequence model of the kind used in REBEL. REBEL fine-tunes BART-large (406M parameters) to emit triples separated by boundary markers, after pretraining on silver Wikipedia triples. We reuse its output format and its pretrained checkpoint as the shared starting point. Each site first fine-tunes locally on its private labeled data.
In each round, the server sends a batch of unlabeled proxy passages. For each passage $x$, site $k$ draws $M$ sampled decodings. It reports every distinct triple $f$ together with its confidence $$c_k(f \mid x) = \frac{1}{M}\sum_{m=1}^{M} \mathbb{1}[f \in S_k^{(m)}(x)],$$ where $S_k^{(m)}(x)$ is the set of triples parsed from the $m$-th sample. This is a Monte Carlo estimate of the probability that $f$ appears anywhere in the model's output. Because membership is counted rather than position, the estimate is unaffected by the order in which the teacher happens to write facts.
The server first canonicalizes triples. It normalizes entity strings and, when sites use different relation names, maps relation strings into a shared surface vocabulary. It then forms a consensus confidence $\bar c(f \mid x) = \sum_k w_k\, c_k(f \mid x)$, where $w_k$ is a site weight. We test two choices for $w_k$: weighting by data size, and weighting by the site's agreement with the others on a held-out slice. The server keeps triples with $\bar c$ above a threshold $\tau$ and trains the student on these consensus sets with the permutation-sampling set loss of Set Learning. Each kept triple is weighted by $\bar c$. The student is sent back as the next round's initialization, as in FedED.
Three design questions get explicit study. The first is the proxy pool. FedED needs an unlabeled in-domain proxy. We compare a real public in-domain pool with synthetic passages produced by the SynthIE pipeline, to measure how much the method depends on having good proxy text. The second is grounding. Where a site's schema is closed, we optionally constrain the student's decoding with the prefix-trie method of GenIE. GenIE uses a two-level constrained beam search over trees of valid entity and relation names, which prevents invented identifiers. Here the trie is built from the union of relation names that sites report, rather than from a single central catalog. The third is document-level text. Long documents make triple lists very long, so on document-level benchmarks we replace entity strings with numbered symbols assigned in order of appearance, following DORE. Consensus is then computed over symbol triples aligned through the shared proxy document.
Two kinds of prior work motivate the proposal. Distantly Supervised Relation Extraction in Federated Settings shows that sites gain by exchanging only small summaries: one confidence score per entity pair is enough for the server to pick a reliable sentence across sites. Our per-fact confidences extend that idea from denoising to distillation. Contrast with Major Classifier Vectors and FedLCC show how much skewed label distributions hurt federated relation extraction. They fix it by aligning classifier vectors or pooled pair embeddings, mechanisms that the file notes are inapplicable to a decoder that has no class slots. Fact-level voting handles label skew differently. A relation that appears at only one site still reaches the consensus set through that site's confident votes, provided the weighting does not drown it out. The experiments test whether it does.
Baselines
The naive reference points come first. The first is local-only training: each site fine-tunes REBEL on its own data and is evaluated on its own test split and on the pooled test set. The second is zero-shot pretrained REBEL. The upper bound is centralized fine-tuning on pooled data, the setting in which REBEL reports 75.4 strict F1 on CONLL04 and 92.0 on NYT. All of these use the same checkpoint, the same tuning budget, and the same output format as our method.
The strongest weight-sharing competitor is federated low-rank adaptation applied to the generative extractor. Low-rank adaptation (LoRA) trains small low-rank update matrices in place of the full weights. We follow SLoRA, which trains and averages these adapters across sites, and we also run the robustness-oriented variant in Towards Robust Parameter-Efficient Fine-Tuning for Federated Learning. It is the strongest competitor on accuracy because it shares the whole adaptation signal, not just outputs on proxy text. We also run full-weight FedAvg on BART for reference. For fairness, all federated methods get the same number of rounds, the same local epochs, and the same learning-rate search. We report accuracy against cumulative upload bytes so that no method wins through a different communication budget. No reported result found on Lacuna for federated LoRA on generative relation extraction. "Beating it" means equal or better F1 under the non-IID splits at a lower communication cost. Matching it within one F1 point at more than ten times less upload counts as a partial win, and we would report it as such.
The strongest existing federated relation extraction systems are FedED on i2b2 (75.09 F1, against 72.55 for FedAvg, as reported in FedED) and FedLCC on CDR under non-IID splits (65.60 F1, against 63.01 for FedAvg, as reported in FedLCC). Both are classifiers that are given gold entity pairs. To compare fairly, we run our generative system in a matched relation-classification mode, conditioning on the marked entity pair and generating only the relation, on the same site splits. Beating them means exceeding 75.09 on i2b2 and 65.60 on CDR. In the joint extraction setting, where the classifiers would need a separate entity recognizer, we run them as a pipeline behind a federated entity tagger. This is the configuration that GenIE argues loses about 18% of facts to entity-stage errors.
The ablation that isolates the key ingredient is token-level ensemble distillation, which is FedED's averaging carried over to the decoder. Teachers are forced along a canonical ordering of the consensus output, and the student matches their averaged next-token distributions. It uses the same proxy pool, the same sites, and the same student. A second ablation keeps fact-level voting but trains the student with ordinary cross-entropy rather than the set loss, which separates the effect of aggregation from the effect of the training loss.
Why not just…?
Why not just run federated LoRA on REBEL and stop there? This is the real threat, and it may well win on raw F1 when sites are similar and bandwidth is cheap. The proposal does not assume otherwise. The case for fact-level distillation rests on three settings where weight averaging is structurally disadvantaged. In the first, sites run different architectures or sizes; FIARSE needs dedicated submodel extraction to handle this, whereas fact voting needs nothing. In the second, sites use disjoint relation vocabularies. In the third, sites refuse to release weights or gradients at all, because Panning for Gold and Simple Yet Effective show that such updates leak training text. The heterogeneity experiments below settle the accuracy question. If federated LoRA matches us under model and schema heterogeneity as well, our contribution shrinks to communication cost and attack surface, and we will say so.
Why not just extend FedED by averaging token logits? This is the natural first attempt, and the Set Learning analysis predicts it will fail in a specific way: averaged distributions over differently ordered outputs yield hybrid triples. We measure this directly. We count student outputs that contain a triple asserted by no teacher, and we track how that rate grows with the number of facts per passage. If token averaging matches fact voting even on dense multi-fact passages such as NYT24 and CDR, the core mechanistic claim is wrong.
Why not keep discriminative federated classifiers such as FedLCC and add a federated entity tagger? These methods are well tuned, and on CDR they report 65.60 F1. But they classify gold or predicted pairs into fixed classes. That gives them no natural way to handle a relation one site uses and another lacks, or one entity pair holding several relations, a case that seq2seq decoders handle naturally according to Effective Modeling of Encoder-Decoder Architecture. The matched classification-mode comparison and the joint-extraction comparison together show whether the generative route costs accuracy where classifiers are strongest and gains it where they are weakest.
Why not skip federation and train on synthetic data? If SynthIE-style synthetic text with known triples were enough, no site would need to participate. We include a synthetic-only student trained with the same compute. A small gap between it and the federated student would mean private data adds little, and that would be an important negative result.
Proposed Experiments
The main testbed is 2010 i2b2/VA, which comes naturally split across three medical centers, using the protocol and proxy setup of FedED. For document-level extraction we use CDR and GDA, with the structural-entropy non-IID partition of FedLCC. For overlapping-relation stress tests we use NYT24, where WDec reports 0.817 F1 centrally. We also use CONLL04 and NYT under synthetic label-skew splits, in which each site holds only a subset of relation types and uses its own names for shared relations. Metrics are strict triple F1 (micro, plus macro over relations to expose rare-relation loss), cumulative upload bytes, and the hybrid-triple rate defined above.
Four checks decide the proposal. The first is fact voting against token-level distillation on passages binned by fact count: the claim predicts a gap that widens with density. The second is fact voting against federated LoRA under three regimes, namely natural site splits, heterogeneous model sizes (BART-base at some sites, BART-large at others), and disjoint relation vocabularies. A win only in the latter two would still support the heterogeneity argument. The third is leakage. We plant canary facts that exist only in one site's private training set and appear nowhere in the proxy pool. We then measure how often the federated student emits them, both under our protocol and under federated LoRA, using the extraction procedure of Simple Yet Effective as the attack. The fourth is sensitivity to the proxy pool, comparing real in-domain proxy text with SynthIE-generated text, which shows whether the method still works when good public text is scarce. We would abandon the proposal in that setting if the method falls below local-only training once the proxy is synthetic.
References
- FedED: Federated Learning via Ensemble Distillation for Medical Relation Extraction
- Set Learning for Generative Information Extraction
- REBEL: Relation Extraction By End-to-end Language generation
- GenIE: Generative Information Extraction
- DORE: Document Ordered Relation Extraction based on Generative Framework
- Distantly Supervised Relation Extraction in Federated Settings
- Contrast with major classifier vectors for federated medical relation extraction with heterogeneous label distribution
- Federated Document-Level Biomedical Relation Extraction with Localized Context Contrast
- SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models
- Towards Robust Parameter-Efficient Fine-Tuning for Federated Learning
- FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extraction
- Panning for Gold in Federated Learning: Targeted Text Extraction under Arbitrarily Large-Scale Aggregation
- Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Models
- Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction