Domain-Aware Poisoning Defenses for Private Federated Domain Adaptation

Federated domain adaptation lets several organizations, each holding labeled data from its own setting (a hospital's scanner, a satellite's sensor, a photo style), help a new party whose data is unlabeled and looks different. The standard tool is a domain discriminator: a small classifier that tries to tell which setting a feature vector came from. The shared feature extractor is trained to fool it, so features stop encoding "where the data came from." To keep updates private, one can split trust between two servers that are assumed not to share their secrets. Each server then sees only masked or encrypted pieces, and poisoned updates are filtered by comparing clients' updates geometrically. This note argues that these two ideas conflict in a specific, predictable way. In adversarial adaptation, honest clients from different domains send discriminator updates that point in nearly opposite directions by construction. A poisoning filter that pools all clients will therefore flag the minority domain as an attacker, and the minority domain is often exactly the target party the system exists to help.

We propose to measure this failure and fix it with domain-grouped robust aggregation run under a two-server encrypted protocol. Outlier filtering is done separately within each domain group, using only inner products, norms and means. These are the operations that existing two-server encrypted aggregation already supports. The models are then combined across groups by a plain secure sum. The claim is narrow and testable. Grouped filtering should keep honest out-of-domain clients that pooled filtering wrongly removes. It should also still catch poisoners, including one attack specific to this setting: a client that pretends to belong to the other domain.

Why the obvious idea is not the right one

The obvious combination is to run the adversarial game itself inside a two-server cryptographic protocol, evaluating the discriminator on encrypted features. The research file lists this as an open gap. Both Janus (two-server verifiable secure aggregation with constant client cost) and AegisFL (two-server encrypted Byzantine filtering built from inner products, L2 norms and means) support only linear and simple geometric operations.

A skeptical reviewer would reply that the game does not need to be encrypted. If each client keeps its features local, it can compute the gradient of the discriminator loss on its own batch, with its own domain label, against the current shared discriminator. Because the loss is a sum over samples, the sum of these client gradients equals the centralized gradient. The feature extractor's reversed gradient, the trick from Domain-Adversarial Training of Neural Networks, is computed locally as well. Secure summation of gradients therefore already protects the minimax game.

We think this objection is correct, so we do not propose encrypted discriminator evaluation. This differs from Federated Adversarial Domain Adaptation (FADA), whose central server holds the global discriminator and sees feature-level information in plaintext. It also differs from PrADA, which protects vertically partitioned tabular data with homomorphic encryption and multi-party computation, and does so with three parties and accepted label leakage.

Once the game reduces to secure summation, two problems remain open, and they are where this proposal lives. The first is poisoning robustness under domain shift, the tension the file flags between AegisFL-style filtering and FADA-style heterogeneity. The second is that a discriminator updated with several local steps on single-domain data collapses toward predicting the client's own label.

Core Claim

Take a discriminator whose final layer is logistic, with weights acting on a hidden representation $h$. Let $p$ be its predicted probability that a sample is from the target, and let $d \in \{0,1\}$ be the sample's true domain. The gradient of the loss with respect to those weights is

$$g = (p - d)\,h .$$

Near the alignment the method aims for, $p \approx 0.5$. Source samples then give roughly $+0.5h$ and target samples roughly $-0.5h$. When features are well aligned, so that $h$ looks similar across domains, honest source and target gradients have cosine similarity close to $-1$.

Cosine- and distance-based filters, such as the clustering filter AegisFL runs under encryption, will therefore treat whichever domain has fewer clients as anomalous. We predict this will happen with no attacker present, and it should worsen as alignment improves.

Grouping removes the artifact. Within a domain group, honest gradients share a sign, so ordinary outlier geometry applies again. The same grouping also exposes a domain-specific attack. A malicious source client that submits a target-labeled (sign-flipped) discriminator gradient is hard to detect in a pooled filter, because it looks like a legitimate target client. Inside the source group it is an obvious anti-correlated outlier.

Research Program

We will build a federated adversarial adaptation pipeline in which source clients and one or more target clients each hold a local feature extractor copy, a label head (sources only), and the shared discriminator. Each round has two parts. Clients take several local steps on the feature extractor and label head against a frozen discriminator. They then take exactly one discriminator gradient step at the current global discriminator, which avoids single-label collapse. We will ablate this split against multi-step local discriminator training.

Updates are aggregated with two non-colluding servers in the style of AegisFL. Clients encrypt their updates. Each client's domain group is provided as a secret-shared one-hot indicator, so group-wise means are weighted sums, which are linear and therefore supported. The servers compute within-group inner products and norms against the group mean and apply AegisFL's filtering rule inside each group. Only accept/reject decisions and the final secure sum are revealed.

Where the source/target role is public, as in FADA's designated target node, the indicator can be public. Hiding finer per-source-domain membership is the case that needs the secret-shared form.

We will state the leakage model explicitly. The servers learn which clients were rejected and the aggregate update. If a group contains a single client, robust filtering is undefined for that group. In that case we treat the lone target as trusted, since it is the party requesting adaptation, and we report this as a limitation.

The poisoning-plus-privacy audit in Privacy-Preserving Federated Learning Scheme with Mitigating Model Poisoning Attacks constrains the design. It found that two-server schemes leak through their defenses. We will therefore enumerate what the filter decisions reveal about domain membership. FADA's mutual-information disentanglement and its attention weighting by cluster-gap statistics are out of scope. Both need non-linear computation over features, which we do not attempt to encrypt.

Baselines

The naive reference is federated source-only training, which aggregates source models with no alignment. On Digit-Five, FADA reports 53.5% average accuracy for this setting Federated Adversarial Domain Adaptation.

The strongest existing adaptation method is plaintext FADA. It reports 73.6% average on Digit-Five, 87.1% on Office-Caltech10 with ResNet-101, and 30.3% on DomainNet. We will rerun it with the same backbone, client partition and round budget, since our pipeline drops its disentanglement module. Reaching FADA's clean accuracy within a small margin, while adding privacy and robustness, is the bar for the clean setting. Exceeding it is not.

Our federated adversarial pipeline with Janus-style secure summation and no filtering is the "privacy but no defense" point. Its overhead and clean accuracy should match the plaintext version, as Janus reports for ordinary training (93.18% vs. 94.1% on MNIST).

The strongest existing defense is AegisFL's encrypted filter applied to the pooled update vector, exactly as published. On HAR it reports roughly 90% accuracy under heavy untargeted attack, against 24% for FedAvg. No reported result was found on Lacuna for any Byzantine-robust aggregator on Digit-Five, Office-Caltech10 or DomainNet under domain adaptation, so we will produce that number ourselves.

Beating AegisFL means three things, all on the same attack and tuning budget, with the filter threshold tuned only on source-domain validation data for both methods. First, a lower false-rejection rate for honest clients in the minority domain. Second, higher target accuracy under no attack. Third, equal or better target accuracy under attack.

Two ablations isolate the key ingredient. Grouped filtering run in plaintext separates the effect of grouping from any numerical effect of encryption. Grouped filtering applied only to discriminator parameters, with pooled filtering on the rest, tells us whether the anti-correlation problem is confined to the discriminator.

Why not just…?

Why not just use DP-SGD on client updates instead of two servers? Private Federated Learning with Domain Adaptation shows that heavily noised federated training falls to 76.9% on a heterogeneous spam benchmark, below a local-only model at 81.2%. Outis reports local-DP error two orders of magnitude above its two-server design. DP also does nothing against poisoning, which is our main question. We will still include a DP-noised variant so the privacy-accuracy trade-off is visible rather than assumed.

Why not just run plaintext FADA on a trusted server? If a trusted aggregator exists, this is simpler and probably more accurate. Our proposal is only useful when no single party may see per-client updates. We report plaintext FADA as the reference to show how much the privacy and robustness cost.

Why not just apply AegisFL unchanged? This is the real threat. If honest gradients across domains are less anti-correlated than our derivation predicts, because the hidden representations differ strongly across domains early in training, then pooled filtering may misfire too rarely to matter. The clean-setting experiment below settles this directly. We will log cosine similarity between source and target discriminator gradients over training. If pooled false rejections stay near zero, the grouped method has no reason to exist, and we will say so.

Why not just use coordinate-wise median or trimmed mean, which tolerate heterogeneity better? AegisFL notes that sorting and comparison are intractable under its packed encryption, so these rules are not available in the two-server encrypted setting. We will run them in plaintext only, as an upper reference on robustness.

Why not just leave the discriminator unfiltered and defend only the classifier? A poisoner can then freely invert the alignment signal, either by sending sign-flipped discriminator gradients or by inflating their norm. Our discriminator-flip attack tests exactly this.

Proposed Experiments

We use Digit-Five, Office-Caltech10 and DomainNet with FADA's leave-one-domain-out protocol, with each source domain split across several clients. We vary the client ratio so that the target group is the minority, the majority, or balanced. HAR is used as a cross-check against AegisFL's published setting. Metrics are target accuracy, per-group honest false-rejection rate, attack detection rate, client computation time, and communication per round relative to Janus-style summation.

There are four attacks: untargeted large-norm updates, label flipping on source classifiers, discriminator sign-flip (a source client posing as target), and alignment sabotage through scaled reversed gradients. The fraction of malicious clients ranges from 0 to 30%.

The first decisive check is the clean run. With no attackers and the target as the minority group, pooled AegisFL filtering should reject target clients at a rate that grows over training, tracking the measured source-target gradient cosine. It should also lower target accuracy relative to unfiltered secure summation. Grouped filtering should not. If this prediction fails, the core claim is false.

The second check is the discriminator sign-flip attack. Grouped filtering should detect it at substantially higher rates than pooled filtering. Unfiltered summation should lose most of the gain over the 53.5% source-only floor on Digit-Five.

The third check is the discriminator-update ablation: one global discriminator step per round versus several local steps. It tests whether local-step collapse is real in practice, since that collapse is the stated reason for our round structure.