Entity-Type Modules for Federated NER Across Silos That Label Different Things

Hospitals, labs, and archives that want to train a shared entity extractor rarely annotate the same entity types. One corpus marks only chemicals, another only diseases, a third only genes. When these silos train one shared tagger with federated learning (each site trains on its own data and sends only model updates to a server, which averages them), a hidden contradiction appears. At the chemistry site, every disease mention is labeled "not an entity" simply because nobody annotated diseases there. Averaging such updates teaches the shared model to suppress the very types the other sites are trying to teach it. We propose to give each entity type its own small module. A module learns only from sites that actually annotate its type. Every module is downloaded and run at every site, so modules can consult one another's internal signals when deciding ambiguous cases like whether "BRCA1" names a gene or a disease.

The claim is narrow and testable. Federated sequence labeling under disjoint annotation fails mainly because unlabeled entities are treated as true negatives. A type-modular design fixes this by construction, and it also does three things a single shared model cannot do cheaply. It lets a new type join without retraining or distilling the old ones. It lets a site download only the modules it needs. And it can recover CollaboNet-style cross-type disambiguation without any site ever seeing another site's text. We take seriously that the first benefit might come from a loss-masking trick on a monolithic model. Much of the experimental design exists to separate that effect from the modularity itself.

What exists and what is missing

Federated named entity recognition (NER) is known to work when every client shares one schema. An in-depth evaluation of federated learning on biomedical natural language processing for information extraction runs FedAvg and FedProx on eight biomedical corpora and finds that federated BioBERT matches centralized training on BC4CHEMD and 2018 n2c2. It also finds that federated models beat 20-shot prompting of GPT-4, PaLM 2, and Gemini Pro. That paper motivates our choice of backbone, but every client in it uses the same fixed label set. FedNER: Privacy-preserving Medical Named Entity Recognition with Federated Learning handles platform differences by sharing the lower encoder layers while keeping each platform's BiLSTM-CRF top layer private. This absorbs annotation-style differences. The cost is that a site can never extract a type that only another site labels, which is exactly the coverage gap we target.

Federated Incremental Named Entity Recognition is the closest work. It studies clients that meet new entity types over time, where old types show up as unlabeled background. Its LGFD method protects the single shared model with SVD-based distillation, a contrastive loss guided by pseudo-labels, and an entropy-based monitor that detects when the task has changed. Our approach differs in mechanism. LGFD guesses labels for background tokens and then fights the resulting drift. We remove the conflict structurally: a module never receives a negative gradient from a site that did not annotate its type. We expect this to matter more as the number of types grows, because pseudo-label noise compounds over many types.

CollaboNet: collaboration of deep neural networks for biomedical named entity recognition supplies the collaboration mechanism we reuse. It trains one tagger per entity type and feeds each "target" tagger the max-pooled hidden states of "collaborator" taggers. This raises macro F1 from 84.07% to 85.15% across five corpora and cuts JNLPBA bio-entity errors from 49.3% to 15.8%. All of this is centralized. Our observation is that collaboration does not require co-located data, only co-located modules. Once every module's parameters are downloaded, each site can compute collaborator states on its own text.

Personalized Federated Learning via Heterogeneous Modular Networks (FedMN) supplies the communication argument. Its clients route through a shared pool of blocks with a Gumbel-Softmax hypernetwork and transmit only the blocks they activate, which roughly halves traffic on FEMNIST. FedMN works only on image classification, and its routing is learned. It also reports that this routing adds instability and overhead. We replace learned routing with routing fixed by the entity schema, since the schema already tells us which module owns which type.

Two vision papers constrain the design. Federated Learning with Bilateral Curation for Partially Class-Disjoint Data and Federated Learning under Partially Disjoint Data via Manifold Reshaping show that clients seeing few classes collapse their features. They fix this with a fixed equiangular classifier or with covariance and prototype losses. Both assume a missing class is absent from the client's data. In NER the missing class is present but mislabeled as background, and background makes up most tokens. Their remedies therefore do not address the core error here, and we treat them as related evidence rather than as the method.

Core Claim

The proposed system has three parts. The first is a shared encoder (BioBERT) trained with FedAvg. The second is one lightweight module per entity type: a low-rank adapter plus a begin/inside/outside tagging head. The third is a small collaboration layer in each module that reads the pooled hidden states of the other type modules, as in CollaboNet.

Let $T_c$ be the set of types that site $c$ annotates, and let $\ell_k(x_i)$ be module $k$'s token-level tagging loss on sentence $x_i$, with that site's gold labels collapsed to type $k$ versus "not $k$". The local objective is

$$ \mathcal{L}_c = \sum_{i \in D_c} \sum_{k \in T_c} \ell_k\big(x_i;\, h_{-k}(x_i)\big), $$

where $D_c$ is site $c$'s data and $h_{-k}$ denotes the hidden states of all other modules, used as inputs but not updated by this term. Module $k$ is averaged at the server only over sites with $k \in T_c$. This one rule means a site's "outside" tokens penalize only the types it actually labels.

A new type is added by creating a new module. Existing modules keep their parameters, so there is nothing to forget except through the shared encoder. We will measure that drift directly. Overlapping predictions across modules are resolved at inference by module confidence. Nested spans are allowed where the schema permits them.

Research Program

We will build the system as a separable harness. The federated communication loop, the per-type modules, and the evaluation code sit behind fixed interfaces, following the lesson of DeepDIVA: A Highly-Functional Python Framework for Reproducible Experiments. With that separation, every baseline below runs on identical data splits and the same aggregation code.

The analysis has three threads. The first measures how much of the gain comes from loss masking alone versus module isolation, as a function of the number of entity types. The second checks whether federated collaboration recovers CollaboNet's reduction in cross-type errors. The third counts the bytes each site transmits per round, and how many shared-encoder parameters change, when a type is added.

Baselines

The naive reference point is single-site training, where each site trains BioBERT only on its own types. It sets the floor that federation must beat. FedNER reports that this kind of baseline reaches a strict F1 of 63.86 on CADEC, against 65.16 for federated training. The second naive point is monolithic FedAvg with one shared head over the union of types, where every unlabeled token is a negative for every type. This is the failure we claim exists, and it must score worse than masking for the story to hold.

The strongest simple competitor is monolithic FedAvg with masked loss: the same encoder and one shared head, but the outside label at site $c$ counts as negative only for types in $T_c$, implemented as partial-label marginalization. This baseline isolates whether modularity matters at all.

FedNER's shared-encoder, private-top design is run as published but with a BioBERT bottom, so that the backbone is equal across methods.

For the incremental setting, the strongest existing method is LGFD from Federated Incremental Named Entity Recognition. It reports Micro-F1 "nearly 9% higher than the strongest baseline" on I2B2, with 10 initial clients and 4 clients joining at each step. No absolute I2B2 or OntoNotes F1 is recorded in our file, so we will rerun LGFD under its own protocol before comparing.

Centralized pooled training with masked loss, and centralized CollaboNet, serve as upper references. CollaboNet's reported macro F1 of 85.15% across its five corpora is the target, though our splits will differ and the comparison is only indicative.

Fairness rules apply to all methods. Every method gets the same BioBERT initialization, the same number of rounds and local epochs, and the same hyperparameter search budget per method. Trainable parameter counts are matched by sizing the monolithic head and adapters to equal the total module budget.

Ablations isolate the key ingredients. We remove the collaboration inputs, giving independent modules. We give modules naive outside-token negatives, which tests modules without masking. We freeze the encoder after the first stage, which tests drift.

Beating the masked monolithic baseline means a statistically significant gain in full-schema F1 over five seeds, or a matched F1 with clearly lower forgetting and communication as types are added. Gains appearing only at large type counts would still count, and would define where modularity is worth its cost.

Why not just…?

Why not just mask the loss in one shared model? This is the real threat. If the masked monolithic baseline matches our system at every type count, then the contribution reduces to a known partial-label loss, and we will say so. We expect modularity to separate from it in three places: as types are added sequentially, where adapters need no distillation; at 18-type OntoNotes scale, where a single head shares capacity among many disjoint gradients; and in communication. The type-count sweep settles this directly.

Why not just use LGFD's distillation and pseudo-labels? LGFD was built for the incremental version of this problem and reports strong gains. It works by guessing labels for background tokens, and those guesses get noisier as the number of types grows. Our incremental OntoNotes and I2B2 runs with randomized type orders compare forgetting curves directly. If LGFD's curve stays flat at 18 types, our structural argument weakens.

Why not just port the equiangular-classifier or manifold-reshaping fixes? FedGELA and FedMR address classes absent from a client's data, not classes mislabeled as background. FedGELA itself names extension beyond classification as open. We will include a FedMR-style prototype loss added to monolithic FedAvg as a secondary baseline, mainly to confirm that it does not address mislabeled background.

Why not just prompt a large commercial model, or pool de-identified data? The federated biomedical evaluation found federated BioBERT beating 20-shot GPT-4, PaLM 2, and Gemini Pro on these extraction tasks. Clinical privacy rules often block both sending records to external APIs and pooling the data. We keep centralized pooling as an upper reference, not as a deployable option.

Proposed Experiments

The main experiment treats each of CollaboNet's corpora as a site: NCBI-Disease, BC4CHEMD, BC2GM, and JNLPBA, plus BC5CDR split into a chemical-only site and a disease-only site. This creates naturally disjoint schemas. The decisive check uses the fully annotated BC5CDR test set. If disease extraction on chemistry-site text is suppressed under naive FedAvg and recovered under masking and modules, the background-collision effect is shown directly. We also report the cross-type error ratio, with CollaboNet's 49.3% to 15.8% reduction on JNLPBA as the reference.

Scaling and incremental experiments split I2B2 (16 types) and OntoNotes 5.0 (18 types) across clients by type, following LGFD's protocol of 10 initial clients plus 4 per step. We sweep the number of types and record final F1, forgetting per type, bytes per round, and encoder drift.

A realism check on CADEC, ADE, and SMM4H follows FedNER's platforms, with each platform assigned a different subset of drug, adverse-event, and symptom types. FedNER's 65.16 strict F1 on CADEC is the reported reference point.

The proposal fails if the masked monolithic baseline equals the modular system on F1, forgetting, and communication at all type counts. It also fails if federated collaboration gives no cross-type error reduction over independent modules.