Hospitals, labs, and companies that want to pull structured facts out of text rarely agree on what to annotate. One site labels drug–adverse-event links, another labels gene–disease associations, a third labels both but not treatment relations. Federated learning is the standard way to train one model across such sites without pooling their private text: each site trains locally and sends only model updates to a coordinator, which averages them. Standard federated training assumes every site uses the same label set. When sites use different label sets, the training breaks in a specific way. A pair of entities that site A marks as "no relation" may be a real relation under site B's schema, so each site actively teaches the shared model to suppress relations it never annotated.
This note argues that relation-first extraction removes most of that conflict almost for free. A relation-first extractor is a model that first decides which relation types a sentence expresses and only then extracts the entities that fill each relation. Because it has no catch-all "no relation" class over entity pairs, its training loss splits cleanly into one term per relation type. A site's loss can be restricted to the relations it annotates, and the coordinator can average each relation's parameters only over the sites that annotate it. We propose to build and test this schema-aware federated relation-first extractor. The claim is that it recovers most of the accuracy of centralized training on the union of all schemas, and that plain federated averaging of the same model does not.
Core Claim
Relation-first models such as the cascade dual-decoder of A Cascade Dual-Decoder Model for Joint Entity and Relation Extraction and the template-filling model RFBFN factor the output into two parts. The first is a relation detector that makes a yes/no decision for each relation type in the sentence. The second is an argument decoder that extracts spans conditioned on a relation. Under federated training with partially overlapping annotation schemas, the naive version of this factorization is fragile. A site with no examples of relation $r$ pushes $r$'s detector output toward "absent" on every sentence. The cascade then makes this failure permanent, because the relation-first papers themselves identify that a relation the detector misses never reaches the entity decoder (Cascade Dual-Decoder; Bio-RFX).
The same factorization offers a fix. Each client $k$ holds a schema $S_k$, the set of relation types it annotates. Client $k$ uses the per-relation detector loss
$$\mathcal{L}_k = \sum_{r \in S_k} \ell_{\text{det}}(r) + \sum_{r \in S_k} \ell_{\text{arg}}(r),$$
where $\ell_{\text{det}}(r)$ is the binary detection loss for relation $r$ and $\ell_{\text{arg}}(r)$ is the argument-extraction loss on sentences where $r$ is present. Relations outside $S_k$ are treated as unknown rather than negative. On the coordinator side, parameters specific to relation $r$ are averaged only over the clients whose schema includes $r$. These parameters are the detector head for $r$ and $r$'s conditioning embedding in the argument decoder. The shared encoder and the shared parts of the argument decoder are averaged over all clients as usual. The prediction is that this relation-aware aggregation, not relation-first architecture alone, closes the gap to centralized training.
Relation to Prior Work
Four papers combine federated learning with relation extraction, and all four either assume given entities or evaluate entity and relation tasks separately. FedED classifies relations between pre-marked entity pairs and distills client predictions over a public proxy set. Its authors name joint entity–relation extraction and non-identical client data as future work. Lazy MIL denoises distant supervision across sites by sharing top instance scores for known entity pairs, and it assumes identically distributed splits. FedLCC handles drift in document-level relation extraction with a contrastive anchor to the global model, again over given mentions. Zheng et al.'s evaluation benchmarks FedAvg and FedProx on named-entity recognition and relation extraction as separate tasks. That evaluation supplies two useful constraints. FedProx helps only slightly and is sensitive to its regularization weight. Performance also drops as the same data is spread over more clients.
These papers motivate the setting, but none of them varies the label schema across clients, and none trains an end-to-end triple extractor. The relation-first papers supply the model: Cascade Dual-Decoder, RFBFN, CLFM (a relation decoder plus attention that filters tokens by relation), and Bio-RFX (relation classification followed by question-style span extraction in biomedical text). All of them are trained centrally on one schema. We reuse their architecture and change only how the loss is masked and how parameters are aggregated.
Research Program
We will build one relation-first backbone and train it under several federated regimes. The backbone is a cascade dual-decoder with independent sigmoid relation heads and a relation-conditioned span decoder, on BERT for general text and SciBERT for biomedical text. We choose the cascade model over RFBFN because it avoids hand-written templates, which would otherwise have to be coordinated across sites.
The study has three parts. First, we will build schema-heterogeneous partitions with a controllable overlap parameter. At one extreme each relation type belongs to exactly one client. At the other, all clients share all types. In between, a chosen fraction of types is shared. We will also vary the relation frequencies across clients independently of schema.
Second, we will measure the failure directly. For each relation, we will record how often the global detector misses it when few clients annotate it. We will also record how much of the triple-level error is explained by detector misses versus argument errors, a diagnostic the relation-first cascade makes easy to compute.
Third, we will test what the argument decoder shares. If conditioning on a relation makes span extraction largely schema-independent, then clients that never annotate $r$ should still improve extraction for $r$ through the shared encoder and decoder. We will measure this by withholding clients and observing the effect on relations they never annotate.
Baselines
The naive reference is local training: each client trains its own relation-first model on its own schema, and at test time the models' predictions are unioned. This shows what collaboration buys. The upper reference is centralized training of the same model on pooled data with the union schema. For context, RFBFN reports 93.6 F1 on NYT and 90.4 F1 on WebNLG, and Cascade Dual-Decoder reports 90.6 exact-match F1 on NYT. Our own centralized run will be the reference we compare against.
The main federated competitor is FedAvg on the identical relation-first model, with missing relations treated as negatives. This is what a practitioner would do by default. The key ablation keeps the schema-masked loss but uses plain averaging over all clients, which separates the masking step from the relation-aware aggregation step. A second strong competitor is FedProx on the same model, tuned over the same small grid of regularization weights for every method, since Zheng et al. found it helps modestly on non-identical data.
The entity-first competitor is a federated joint model that finds entity spans and then classifies all entity pairs with per-relation sigmoid outputs. It is given the same schema masking and the same aggregation, so that the only difference is the extraction order.
No reported result was found on Lacuna for federated joint triple extraction on NYT, WebNLG, or any biomedical corpus. The closest federated number, FedLCC's 65.60 F1 on CDR against FedAvg's 63.01 (FedLCC), is for relation classification over given mentions and is not directly comparable.
All methods will share the backbone, communication rounds, local epochs, client sampling, and hyperparameter budget. Beating the baselines means two things. At low schema overlap, the proposed method must close at least half of the triple-F1 gap between FedAvg and centralized training. It must also cut the detector's miss rate on relations annotated by a single client, and its advantage over the masked-loss ablation must hold across three seeds.
Why not just…?
Why not just mask the loss and keep plain FedAvg? This is the cheapest version of the idea, and it may deliver much of the gain. Masking stops clients from suppressing relations they do not annotate. However, averaging a relation's head over clients that never train it still shrinks that head toward its initial values, by a factor of roughly the fraction of clients holding the relation. The ablation described above measures exactly this. If it matches the full method, we will report masking alone as the finding and drop the aggregation change.
Why not just use an entity-first pair classifier with the same masking? This is the most serious threat. Per-relation sigmoid outputs over entity pairs split the loss by relation just as cleanly. The expected differences are twofold. The entity-first model's span finder must still be trained with a single notion of "is an entity," which conflicts across schemas that annotate different entity types. It also sees quadratically many negative pairs, and the relation-first papers attribute much of their overlap robustness to avoiding those pairs: RFBFN reports 90.4 F1 on WebNLG against 42.9 for the all-pairs GraphRel. The matched comparison will settle the question. If the entity-first model ties, the contribution is the schema-aware protocol, not the extraction order, and we will say so.
Why not just distill into a central model, as FedED does? Distillation requires unlabeled public text from the same domain at the coordinator, which FedED itself flags as often unavailable. It is also unclear how to combine structured outputs from teachers with different schemas, such as sets of spans for each relation. We will include a distillation variant only on NYT and WebNLG, where public text exists, as a check on communication cost.
Why not just prompt a large language model with the union schema? Zheng et al. found that few-shot prompting of GPT-4, PaLM 2, and Gemini Pro trailed federated domain models on biomedical extraction. Bio-RFX reports that it outperformed GPT-4 on nested biomedical relations. Sending clinical text to external APIs is also the privacy problem federation exists to avoid. We therefore treat prompting as a reference point, not a core baseline.
Proposed Experiments
On NYT (24 relations) and WebNLG (more than 200 relations, many overlapping triples), we will build 4-, 8-, and 16-client partitions at schema overlaps of 0, 25, 50, and 100 percent. Test sets use the union schema, and we report exact-match triple F1 overall, for sentences with overlapping triples, and for relations held by only one client.
For the biomedical setting, we will federate across corpora whose annotation schemas genuinely differ, including sentence-level BC5CDR, DrugProt, and DrugVar, all used by Bio-RFX. Each corpus acts as a client, and evaluation is on each corpus's own test set with a model that must also avoid spurious relations from the other schemas. If access permits, the three-site 2010 i2b2/VA data gives a naturally partitioned setting with a shared schema.
The decisive checks are three. The first is the curve of the FedAvg-to-proposed gap against schema overlap: it should shrink to near zero at full overlap, where both methods are identical, and widen as overlap falls. The second is the breakdown of triple errors into detector misses and argument errors: the gain should come mostly from fewer detector misses. The third is the matched comparison with the entity-first model, which decides whether extraction order matters or only the protocol does. We will also report bytes communicated per round. The relation-specific parameters are small, so selective averaging adds little overhead, and we will check that claim empirically.
References
- A Cascade Dual-Decoder Model for Joint Entity and Relation Extraction
- RFBFN: A Relation-First Blank Filling Network for Joint Relational Triple Extraction
- Bio-RFX: Refining Biomedical Extraction via Advanced Relation Classification and Structural Constraints
- FedED: Federated Learning via Ensemble Distillation for Medical Relation Extraction
- Distantly Supervised Relation Extraction in Federated Settings
- Federated Document-Level Biomedical Relation Extraction with Localized Context Contrast
- An in-depth evaluation of federated learning on biomedical natural language processing for information extraction
- CLFM: Contrastive Learning and Filter-attention Mechanism for Joint Relation Extraction
- GraphRel: Modeling Text as Relational Graphs for Joint Entity and Relation Extraction