In chemistry patents, lab protocols and materials papers, the same substance plays different parts in different sentences. Ethanol can be a solvent in one step and a starting material in the next. A material can be an anode in one experiment and a support in another. Deciding which part a mention plays, its functional role, depends on the surrounding operation ("dissolved in", "added dropwise to", "calcined"). The name alone does not settle it. The labeled text that would teach this skill sits in separate institutions: company notebooks, unpublished filings, individual labs. Each annotates under its own label scheme. Federated learning, where a model is trained at each site and only weight updates are pooled, is the natural way to combine them. Existing federated extraction systems, however, keep exactly the role-deciding layers private, because those layers are tied to each site's labels.
We propose to federate the part of the model that turns context into a role, and to do it without forcing sites to agree on labels. Each site trains the shared upper layers with a loss that needs no common vocabulary: mentions its own annotators gave the same role should have nearby representations, and mentions given different roles should be far apart. Each site keeps only a thin private classifier that names the clusters in its own scheme. The bet is that "these two mentions play the same part" carries the same meaning across institutions even when "CATALYST" versus "REAGENT_CATALYST" does not. If so, pooling this signal should help most on the cases that define the problem: identical names in different roles, and rare roles that no single site sees often enough.
Core Claim
When several sites with incompatible role schemas collaborate, a schema-free pairwise objective over contextual mention representations is the right thing to share. It should beat three alternatives. The first is keeping role layers private, as in FedNER. The second is averaging a whole model over a hand-built union label set. The third is the strong default of a domain-pretrained encoder fine-tuned locally at each site. The gains should concentrate on ambiguous surface forms and long-tail roles, not on the frequent, easy mentions.
What the Literature Gives and Leaves Open
Three kinds of evidence motivate the problem.
The first establishes that role is a separate axis from identity. An accurate and precise representation of drug ingredients separates molecular substance from functional roles such as lubricant or glidant in a drug ontology. The WEAVE 2.0 corpus annotates 71 labels that cross seven naming types with reaction roles. Cooking text shows the same pattern, with a pan acting as tool or destination (Understanding the Cooking Process with English Recipe Text).
The second shows how far centralized models get on role labeling. WEAVE 2.0 reaches 0.87 F1 with a patent-adapted SciBERT ("ChemicalBERT") and an attention decoder. ChEMU 2022's best exact-match NER-with-roles system reaches 0.9673 F1 (Extended Overview of ChEMU 2022). ReactIE reaches 81.6% F1 on catalyst/solvent role extraction with weakly supervised generative question answering.
The third points to where these models remain weak. ChEMU 2020 reports roughly a 10-point F1 drop on patents from new sources. WEAVE 2.0 and MELT both document collapse on minority roles. Both symptoms are what data from a single site produces.
FedNER supplies the federated method we build on and argue against. It averages the bottom layers (embeddings and a word-level CNN) across platforms and keeps a BiLSTM-CRF tagger private at each site. It reaches 65.16 strict F1 on CADEC against 63.86 for the best single-platform model, but its entity types are static. For role ambiguity, the private top is precisely where the operation-to-role reasoning lives. FedNER's design therefore shares the least useful part. We keep its private-head idea but move the boundary up, and we replace "synchronize everything below the labels" with an explicit cross-site objective.
Federated Multi-Task Learning (MOCHA) frames each client as a task with a learned task-relationship matrix. It is restricted to convex models and needs a central step to estimate task relations. We borrow the framing that each site is a task and drop the machinery. Photon shows federated language-model pretraining can match centralized pretraining. We treat that as permission rather than novelty: we do not propose a new federated pretraining method.
Two papers constrain the design. ChEMU's finding that pipelines often beat joint end-to-end models warns that naive joint training across heterogeneous objectives can cause negative transfer. InstructUIE loses ground on large individual datasets when 32 datasets are balanced together. Both mean our method must not simply add tasks to a shared model and hope.
Research Program
The encoder starts from a domain-pretrained checkpoint, because Männistö's PEFT study found domain pretraining a stronger predictor of downstream performance than the choice of fine-tuning method. A materials-aware or patent-adapted backbone in the spirit of ChemicalBERT and MELT is therefore the fixed starting point, not a variable.
On top of the encoder sit two shared projections of each mention span. The identity projection is trained where available on naming type (WEAVE 2.0's systematic/formula/trivial distinction). The role projection $r_i$ is trained with a supervised contrastive loss computed entirely inside one site's batch:
$$ \mathcal{L}_k = -\sum_{i}\frac{1}{|P_k(i)|}\sum_{j\in P_k(i)} \log \frac{\exp(r_i^\top r_j/\tau)}{\sum_{m\neq i}\exp(r_i^\top r_m/\tau)} $$
Here $k$ indexes the site, $i,j,m$ index mentions in a batch from that site, $P_k(i)$ is the set of mentions site $k$'s own annotators gave the same role as $i$, and $\tau$ is a temperature. The loss never refers to a label name, so sites with 10, 16 or 71 labels contribute updates to the same parameters.
Schemas differ in granularity, and this creates an asymmetry. When a coarse-schema site calls two mentions different, a fine-schema site would also call them different. When it calls two mentions the same, a fine-schema site might split them. Negatives therefore transfer across granularities and positives do not. We weight positive pairs per site by a factor $\alpha_k$, set from the site's label count and tuned on its validation split. This is a small, testable mechanism, not a heuristic hidden in the training recipe.
Each site also keeps a private linear or attention-decoder head that maps $r_i$ to its own labels. Rounds of local training are averaged with FedAvg over the encoder and both projections. Only weights leave a site. We report this honestly as no formal privacy guarantee, and we include one differentially private arm to measure the accuracy cost.
The analysis asks whether the shared role space actually organizes mentions by function. We embed held-out mentions of surface forms that occur in at least two roles. We then measure whether nearest neighbors share the role or merely the name. We also check whether clusters learned mostly from chemistry sites align with SOFC-Exp slots and cooking roles that the chemistry sites never labeled.
Baselines
The naive reference point is local-only training. Each site fine-tunes the same domain-pretrained encoder with its own head on its own data, with no communication. This is the strongest simple competitor, because the backbone already carries domain knowledge. Every other arm must clear it, and especially on the ambiguity and long-tail subsets, where the hypothesis says collaboration should matter.
The strongest existing federated method is FedNER's shared-private split (FedNER), re-implemented on the same transformer encoder. The lower encoder layers are averaged and the top layers plus tagger stay private. The split point is chosen on validation from three candidates, so it is not handicapped by a single arbitrary choice. Its reported result is 65.16 strict F1 on CADEC versus 63.86 single-platform. No reported result was found on Lacuna for FedNER on any chemical role corpus, so we run it ourselves.
The third baseline is FedAvg over a union label space. We hand-map all schemas into a merged inventory and average one full model. This tests whether the label-free loss beats a careful human alignment.
The centralized pooled multi-task model trains the same encoder with per-dataset heads on all data together. It serves as the non-private upper reference, following WEAVE 2.0's joint-head design. On WEAVE 2.0 the centralized number to compare against is 0.87 F1 (ChemicalBERT plus attention with augmentation, WEAVE 2.0). On ChEMU NER with roles it is 0.957 (2020) and 0.9673 exact match (2022). On ReactIE's role extraction it is 81.6% F1. For SOFC-Exp slot filling the file gives only inter-annotator agreement (about 78.3 F1) and MELT's relative gains, so no reported absolute slot-filling F1 was found on Lacuna for a direct comparison.
Two ablations isolate the key ingredient. The first replaces the pairwise role loss with plain FedAvg of the role projection under private heads. The second replaces it with an identity-contrastive loss that pulls together mentions with the same surface form. That control should help easy mentions and hurt ambiguous ones, which separates "sharing context-to-role reasoning" from "sharing better chemical-name embeddings."
All arms use the same backbone, the same per-site data and splits, the same number of local steps and communication rounds, and an equal hyperparameter search budget per arm on site validation sets. Beating a baseline means a gain on the ambiguous-mention subset and on long-tail macro-F1, with paired bootstrap confidence intervals excluding zero across three seeds, and no loss beyond one F1 point on overall per-site micro-F1.
Why not just…?
Why not just pretrain centrally on public patents and fine-tune locally? This is the most serious threat. ChemicalBERT was built from 230 million lines of public patent text, so much of the domain knowledge federation might supply is already free. Our local-only baseline is exactly this approach. If the federated arm does not beat it on ambiguous mentions and rare roles, the claim fails, and we will report that. The reason to expect a gain is that public pretraining teaches what chemicals are called, not which part they play. Role supervision is scarce at every site. FedNER found that the gap between federated and single-site training widens as local data shrinks, and we test this directly by subsampling each site's labels.
Why not just map everyone's labels into one scheme and run FedAvg? Mapping a 71-label scheme onto a 10-label one throws away distinctions or invents ones the coarse annotators never made. Each new site also requires expert ontology work. We include the union-mapping arm. Our prediction is that it matches us on coarse sites and loses on WEAVE 2.0's fine labels. If it matches us everywhere, the label-free loss is unnecessary and the simpler method wins.
Why not just use FedNER's private-top design with a modern encoder? The design is sound for static entity types. In role labeling, though, it keeps the layers that read governing operations private, so each site's rare roles get no help from other sites. The split-point sweep in our baseline gives this approach every chance. The ablation without the pairwise loss shows whether moving the boundary helps or whether the explicit objective is needed.
Why not just prompt a large language model? Proprietary procedures often cannot be sent to an external API. InstructUIE also reports gpt-3.5-turbo at 18.22 F1 on OntoNotes, which suggests general models do not reliably produce structured extraction. That evidence is dated and comes from a general-domain benchmark. As a reference rather than a competitor, we add a locally hosted open instruction-tuned model, prompted few-shot on the ambiguity subset.
Proposed Experiments
The federation is simulated over public corpora with genuinely different schemas. One site is WEAVE 2.0 (33 patents, 71 labels). Two sites are ChEMU 2020 split by patent source (EPO versus USPTO), which gives non-IID clients under the same schema and reproduces the cross-source drop. One site is ReactIE's 806-reaction corpus (coarse roles). One site is SOFC-Exp (45 papers, 16 slots). A cooking-recipe site is held out entirely, to test whether the role space transfers to a domain with tool/target/destination roles.
The evaluation reports per-site micro-F1 and macro-F1. It also reports F1 on an ambiguity subset: mentions whose normalized surface form appears in at least two roles in that site's training data. Long-tail macro-F1 covers roles below a frequency threshold. Cross-source F1 is measured on ChEMU 2022's added test snippets.
The decisive checks are four. First, the federated pairwise arm must beat local-only on the ambiguity subset and the long tail at every site. Second, the identity-contrastive control must not produce that gain. If it does, what transfers is name knowledge, not role reasoning. Third, the gap to the centralized pooled model should be small relative to the gap between local-only and centralized. Fourth, the advantage over local-only should grow as each site's labeled data is subsampled to 25% and 10%.
A secondary experiment tests the positive-pair weighting. Setting $\alpha_k$ to one for all sites should hurt WEAVE 2.0's fine labels when ChEMU and ReactIE are present, and the tuned weighting should recover that loss. A final arm adds differentially private noise to the averaged updates and traces accuracy against the privacy budget on the ambiguity subset.
References
- FedNER: Privacy-preserving Medical Named Entity Recognition with Federated Learning
- An accurate and precise representation of drug ingredients
- The WEAVE 2.0 Corpus: Role Labelled Synthetic Chemical Procedures from Patents with Chemical Named Entities
- Understanding the Cooking Process with English Recipe Text
- Extended Overview of ChEMU 2022 Evaluation Campaign: Information Extraction in Chemical Patents
- ReactIE: Enhancing Chemical Reaction Extraction with Weak Supervision
- ChEMU 2020: Natural Language Processing Methods Are Effective for Information Extraction From Chemical Patents
- MELT: Materials-aware Continued Pre-training for Language Model Adaptation to Materials Science
- Federated Multi-Task Learning
- The Future of Large Language Model Pre-training is Federated
- InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction
- A Comparative Study of PEFT Methods for Python Code Generation