Telling Minority Perspectives from Noise in Federated Aggregation

When people label subjective content such as hate speech, irony or harm, they often disagree for good reasons: different backgrounds lead to different thresholds. Federated learning trains one model across many devices or institutions without pooling their raw data, so each annotator's labels stay on their own client and only model updates reach the server. That server faces a problem the perspectivist literature never had to solve. A client whose updates point away from the consensus could be careless, adversarial, or a coherent minority voice, and every existing federated aggregation rule treats these three cases the same way. Byzantine-robust filters trim them, quality-weighted schemes pull them toward a global anchor, and fairness reweighting boosts them. Each choice is wrong for at least one of the three cases.

We propose a cheap, local test that separates the cases. A genuine minority perspective is predictable: if a client disagrees with the global model on some items, a small private model fit to half of that client's labels should predict where it disagrees on the other half. Random noise fails this test. The resulting coherence score, a single scalar per client, decides whether a client's deviation is kept as a perspective, learned through a shared generator of perspective-specific adapters, or shrunk away as noise. The claim is that this preserves minority annotators' labelling behavior in a federated model while staying robust to noisy clients, which neither group-debiased nor quality-weighted federated methods can do.

Core Claim

The field already documents erasure in both directions. Majority vote and pooled objectives silence minorities. Toward a Perspectivist Turn in Ground Truthing for Predictive Computing argues this conceptually. MaxMin-RLHF proves that a single reward cannot align sufficiently diverse users. No Preference Left Behind shows reward margins for minority opinions shrinking below zero during preference optimization. On HateXplain, Majority Vote Silences Minority Values finds that hard-label, soft-label and per-annotator-head models all stay about 70% majority-aligned and drop from roughly 80% to 58% accuracy on contested posts. That study also finds annotators who highlight identical rationales yet assign different labels, which is direct evidence that much of the disagreement is principled rather than careless. These papers motivate the proposal.

The federated side constrains it. Trustworthy Federated Label Distribution Learning under Annotation Quality Disparity (FedQual) treats divergent label distributions as low quality and anchors those clients toward the global model. It also assumes per-client quality scores are known in advance, which subjective tasks cannot supply. Rethinking Personalized Reward Modeling … via Group-Debiased Federated Learning (FedGD) counters imbalance by sampling equally from $K$ discovered preference groups. A group of noisy clients would receive the same equal vote, and $K$ must be fixed in advance. Towards Federated RLHF with Aggregated Client Preference for LLMs (FedBiscuit) clusters clients by loss similarity, which mixes noise, scale offsets and real disagreement together. Byzantine defenses such as Holdout SGD and MUDGUARD remove outlying updates by design.

What we add is a criterion that does not depend on how far a client deviates or how large it is. The criterion is whether its deviation is structured. For client $k$, split its local items into halves $A$ and $B$. Fit a small local adapter on $A$ and score

$$c_k = \frac{\mathcal{L}_B(\theta_g) - \mathcal{L}_B(\theta_g + \Delta_k^A)}{\mathcal{L}_B(\theta_g)},$$

where $\theta_g$ is the current global model, $\Delta_k^A$ is the adapter trained on half $A$, and $\mathcal{L}_B$ is the client's loss on half $B$. A coherent minority client has high $c_k$ even when its loss under $\theta_g$ is high. A label-flipping noise client has $c_k \le 0$ however much it deviates. A client that simply agrees with the majority has low loss and little room to gain, and it contributes normally to the shared model.

Research Program

The model has three parts. The first is a frozen pretrained encoder (RoBERTa-base for classification). The second is a shared trainable trunk adapter. The third is a hypernetwork, a small network that outputs the weights of another network: it maps a low-dimensional perspective code to LoRA matrices, which are small low-rank weight corrections added to the frozen model. We borrow this design from Hypernetworks for Perspectivist Adaptation, which generates annotator-specific LoRA weights centrally from annotator IDs. In our version each client's perspective code stays on the client and is never transmitted. The server averages only the trunk and hypernetwork weights. This addresses the privacy concern raised by You Are What You Annotate, whose annotator embeddings cluster along demographic lines, and it avoids FedBiscuit's growing ensemble of adapters and FedGD's fixed $K$.

The coherence score controls aggregation. Each round, client $k$ computes $c_k$ locally, with a cheap one-epoch adapter, and sends a clipped and optionally noised version with its update. The server weights the client's contribution to the hypernetwork by $\sigma(c_k/\tau)$, a soft gate with temperature $\tau$, so coherent deviators shape the space of perspectives. Contributions to the trunk use sample-size weights multiplied by the same gate, so incoherent clients are shrunk. This plays the role of PEBS-style empirical-Bayes shrinkage, but the strength of shrinkage is set by coherence rather than by rating count. That directly targets PEBS's acknowledged failure of pulling rare but genuine extreme raters toward the mean.

The analysis has two parts. We will study when split-half coherence fails. Clients with very few labels give noisy estimates of $c_k$, and coordinated adversaries are coherent by construction. We will also characterize how many labels per client are needed for $c_k$ to separate the cases reliably.

Baselines

The naive reference point is FedAvg with sample-size weighting on majority-vote labels, followed by local fine-tuning. It is the default deployment and the setting in which every erasure finding above was observed.

The strongest existing competitor is FedGD from the group-debiased federated paper. It is the only federated method that explicitly protects small preference groups. It reports 92.35% post-adaptation accuracy in the imbalanced 15/15/5/5 synthetic persona setting and 64.41% on Reddit TL;DR, against 57.06% for FedBiscuit. We will rerun FedGD on our splits with the same backbone and communication rounds. Its $K$ will be tuned on the same validation budget we give our temperature $\tau$. Beating it means matching its minority-group accuracy on clean clients while degrading less when noise clients are injected.

FedQual stands in for the quality-weighting family. It reports KL divergence falling from 0.252 to 0.116 on PIPAL-LDL. Because FedQual needs quality scores we cannot know, we will give it oracle quality in synthetic settings and quality estimated from agreement with the global model in real ones. That estimate is exactly the proxy we expect to erase minorities.

$q$-FFL from Fair Resource Allocation in Federated Learning represents the fairness family. It raises the bottom-10% device accuracy on Vehicle from 43.0% to 69.9%. We sweep $q$ under the same tuning budget. We include a coordinate-wise trimmed mean as the Byzantine family.

Two ablations isolate our key ingredient. The first replaces $c_k$ with raw local loss under $\theta_g$, which is roughly what loss-based clustering and $q$-FFL respond to. The second removes the gate entirely, leaving a plain federated hypernetwork. If raw loss works as well as $c_k$, the predictability idea adds nothing.

For annotator-level F1 on $D_{MDA}$ or $D_{RB}$ in any federated setting, no reported result found on Lacuna. The centralized hypernetwork's annotator-level F1 on those sets also has no reported number in the file. We will therefore run it as the centralized upper bound.

Why not just…?

Why not just use FedGD's equal-per-group sampling? FedGD protects small groups well when every group is legitimate. Its discovery phase, however, assigns clients to whichever expert fits best, so a set of noisy clients becomes a cluster and receives equal sampling weight. Its reported results come only from settings without injected noise. Our noise-injection experiments test this directly. If FedGD holds up under 20–40% noise clients, our gate is unnecessary and we will report that.

Why not just personalize locally and ignore aggregation? This is a real threat for clients with many labels. However, the FedGD paper shows minority clients plateau when initialized from a majority-saturated global model. PEBS finds that naive per-rater fitting needs at least 20 ratings to break even, against 5 with shrinkage. We will sweep labels per client from 5 to 200 and report where local fine-tuning catches up.

Why not just up-weight high-loss clients with $q$-FFL? High loss is what a noisy client produces too, so $q$-FFL should amplify noise most strongly. Its authors also report instability at large $q$ under heterogeneity. The raw-loss ablation tests this mechanism without $q$-FFL's optimizer.

Why not just train on soft labels? On HateXplain, the soft-label model in Majority Vote Silences Minority Values stays about 70% majority-aligned and shows the same 22–28 point drop on contested posts. Population-level distributions also cannot say which annotator holds which view.

Proposed Experiments

We treat each annotator as a client on the perspectivist adaptation sets $D_{MDA}$ (334 annotators), $D_{RB}$ (819) and $D_{EPIC}$ (74), on the many-annotator subsets of TID-8, and on PRISM. PRISM's census-matched demographics let us define minority groups independently of the model. We will also reproduce FedGD's 40-client persona setup with a 15/15/5/5 imbalance. In every setting we inject controlled fractions of three client types: uniform label-flip noise clients, clients that copy the majority, and coordinated adversaries sharing one target flip.

The metrics are annotator-level F1 split by majority versus minority annotators, item-level disagreement correlation, and the rate of majority alignment on contested items. We also measure the area under the curve for $c_k$ separating injected noise clients from real ones. On HateXplain we report accuracy on disagreement posts against the 58% reference.

The decisive check has three conditions. In the synthetic setting, $c_k$ must separate minority personas from noise clients, with AUC well above that of raw-loss scoring. Our method must keep minority-persona accuracy near FedGD's clean 92.35% as noise rises, while FedGD and $q$-FFL fall. On PRISM, demographic-minority raters must get higher coherence scores than injected noise and better calibration than with FedAvg plus fine-tuning. If coordinated adversaries pass the coherence test, as we expect, we will state that our method separates perspectives from noise but is not a Byzantine defense. We will then evaluate it combined with trimmed means.