Auditing Federated Unlearning Across Nested Model Widths

When a client leaves a federated system (many devices training one shared model without pooling their data) and asks for its data to be forgotten, someone has to check that the server actually removed it. Every current way of checking breaks in practice in one of three ways. The first is to publish the before-and-after weight difference as proof, but that difference can be inverted back into the forgotten images. The second is to watch the model's outputs, but outputs can be silenced while the internal features still hold the data. The third is to compare against a model retrained from scratch, which costs exactly what unlearning was meant to save. We propose a check that sidesteps all three. It works in the increasingly common setting where weak and strong devices train nested slices of one network: a phone trains the first quarter of the channels and a server-class machine trains all of them, as in HeteroFL and FjORD.

The idea fits in two sentences. The departing client keeps its own forgotten data and a small held-out set. After unlearning, it runs both through every nested sub-model of the new global model and measures, on-device, how much more its forgotten labels can be read from internal features than its held-out labels. Only a single pass/fail statistic leaves the device. Because a cheating server would have to suppress the forgotten information consistently in every nested slice at once, we conjecture that cheap fakes, such as rewriting only the classifier head or patching the full-width output, become detectable. We also conjecture that a fake which survives the whole ladder costs about as much as honest unlearning. If that holds, the protocol is the first unlearning check for heterogeneous-architecture federated learning that publishes nothing an attacker can invert.

Why this gap, and what already exists

Three recent papers show that the artifacts of federated unlearning leak the very data being deleted. FUIA reconstructs forgotten images from the global-model difference $W^o - W^u$. It also identifies a single erased class with 100% success. DRAUN inverts updates from gradient-ascent-style unlearning, reaching SSIM 0.6407 and PSNR 21.11 on CIFAR-10. Verifiably Forgotten? shows that "proof of federated unlearning" gradient differences are themselves invertible, with an 88% MSE reduction over the prior attack. That paper proposes rotating the difference to preserve its norm while destroying its direction.

These papers motivate our design rule: the proof must not be a weight or gradient vector. All three also assume every client shares one weight shape, and none considers nested widths.

On the verification side, two papers constrain what a check may rely on. An Information Theoretic Evaluation Metric for Strong Unlearning shows the "head distillation" trap: retraining only the final layer passes membership-inference and accuracy tests while the forgotten feature clusters survive. It proposes the Information Difference Index (IDI), a contrastive estimate of how much forgotten-label information remains in intermediate features. Verification of Machine Unlearning is Fragile shows that dishonest providers can forge retraining logs or mimic gradient influence while keeping about 64% F1 on forgotten data, against about 38% for honest retraining.

Two observations shape our method. IDI needs white-box features and a retrained reference model, both unusable in federated learning as given. Any process-based proof can be forged.

The heterogeneous-architecture methods supply the setting. HeteroFL gives clients channel-width fractions $r$. FjORD orders neurons so that narrower sub-models are prefixes of wider ones. Neither paper discusses unlearning. Our proposal differs from all of the above in three ways: the auditor is the data owner, the evidence is a statistic over a ladder of nested sub-models, and the reference is the client's own held-out data rather than a retrained model. We restrict scope to nested-width systems. Logit-exchange systems like FedDF share no parameter slices, so the ladder does not apply to them.

Core Claim

Let $z_r(x)$ be the penultimate features of the width-$r$ nested sub-model of the post-unlearning global model. Let $\hat I(\cdot\,;\cdot)$ be an InfoNCE mutual-information estimate as in IDI. The client computes a per-width gap $g_r$ and passes the server if the largest gap over the width ladder $\mathcal R$ is at most a tolerance $\tau$:

$$ g_r = \hat I\big(z_r(X_f);Y_f\big) - \hat I\big(z_r(X_h);Y_h\big), \qquad \text{pass iff } \max_{r\in\mathcal R} g_r \le \tau . $$

Here $(X_f, Y_f)$ is the client's forgotten data with its labels and $(X_h, Y_h)$ is its held-out data with labels. The held-out set stands in for the retrained reference. A model that never saw $X_f$ should treat forgotten and held-out samples alike, so the gap should be near zero.

We make three testable claims. First, the ladder catches cheating that a full-width check alone misses. The client's influence was written into the low-index core channels, which every narrower sub-model contains, so a server that suppresses only the head or only the full-width path leaves a positive gap at small $r$. Second, the protocol's only transmitted artifact, one statistic, gives near-chance reconstruction under FUIA-, DRAUN- and IGF-style attacks, while published slice-restricted deltas do not. Third, defeating the ladder requires optimizing the forgetting objective jointly over all widths, and we will measure whether that costs roughly as much as honest unlearning.

Research Program

We first build the testbed. We implement federated unlearning on HeteroFL-style and FjORD-style width-nested networks, adapting three families of method to slices.

  1. Retraining from scratch, the gold reference.
  2. Gradient-ascent and gradient-difference unlearning, as attacked by DRAUN.
  3. Historical-update subtraction followed by distillation, as in Federated Unlearning with Knowledge Distillation, restricted to the client's channel slice.

We then build dishonest servers that each pass a known check:

  • Head-only retraining, following the head distillation trap.
  • Output suppression trained against a simulated membership-inference adversary, the game formalized by Di et al..
  • Peripheral-only edits, which change only high-index channels that the departing low-tier client never touched.
  • Gradient-influence mimicry adapted from the Fragile paper.
  • An adaptive cheater that knows the audit and runs output suppression at a subset of widths, or at all widths, under a fixed compute budget.

Second, we measure leakage. We ask whether publishing the slice-restricted delta, the natural proof in nested systems, is more or less invertible than a full-width delta. FjORD notes that ordered dropout behaves like an SVD of linear maps, and IGF uses SVD as its first step. That convergence suggests narrow-slice deltas may be especially invertible. This question was unstudied, and the measurement tells us how much is at stake in not publishing deltas.

Third, we characterize the audit statistic itself. We study how the estimate of $\hat I$ and the variance of the gaps depend on the sizes of $X_f$ and $X_h$. We study how to set $\tau$ for a target false-alarm rate against honest retraining. We study whether the single released statistic leaks membership of other clients' data, which we will test by running membership inference on the statistic's distribution.

Proposed Experiments

We use CIFAR-10 and CIFAR-100 with Dirichlet non-IID splits, and FEMNIST with its natural per-writer split, matching the settings of HeteroFL, FjORD and DRAUN. The architectures are PreResNet18 with HeteroFL widths a–e and ResNet18/CNN with FjORD ordered dropout. Departing clients are drawn from each width tier, because a low-tier client is the hardest case for full-width checks.

The primary verification metric is the auditor's AUROC for separating honest unlearning runs from each cheating family, with retraining defining the honest distribution. We also report the false-alarm rate at fixed $\tau$. The primary leakage metric is SSIM, PSNR and LPIPS of the best reconstruction obtainable from each protocol's published artifact, using FUIA-, DRAUN- and IGF-style attacks run by us. Utility is reported as global and per-tier accuracy after unlearning, and cost as client-side FLOPs for the audit and server FLOPs for each cheater.

Baselines

Trusting the server. The naive reference is to accept the server's unlearning claim with no check. Any audit should beat it by catching cheaters at all; we report how often each cheating family would go undetected.

Backdoor and membership-inference checks. The strongest deployed black-box verification is backdoor trigger collapse combined with membership inference. Federated Unlearning with Knowledge Distillation reports attack success rate falling from about 100% to 0% on MNIST and GTSRB. We run it on the same splits and widths, with the backdoor planted in the departing client's data before training. Beating it means higher AUROC on the head-only and output-suppression cheaters. The IDI paper's head-distillation result predicts this baseline is near chance on those cheaters.

Proof-of-unlearning with orthogonal obfuscation. The strongest competitor that publishes a proof is the gradient-difference proof with Orthogonal Obfuscation from Verifiably Forgotten?, which we apply per slice. It is the most direct prior answer to the leakage problem. We give its attacker the same auxiliary data and inversion-model training budget as attackers against our protocol. Beating it requires both lower reconstruction quality from our released statistic and higher detection AUROC. A norm-preserving rotated proof says nothing about where the information went, so we expect it to fail against mimicry and peripheral-only cheats.

Retraining reference. Retraining from scratch defines the honest distribution and the utility ceiling. It is not a competitor on cost.

Full-width ablation. The key ablation is the same client-side statistic computed only at full width, $\mathcal R=\{1\}$. This is essentially IDI moved onto the client with a held-out reference. If it matches the ladder's AUROC, the nested-width ingredient adds nothing, and the contribution shrinks to client-side IDI. A second ablation replaces the held-out reference with a retrained model, to measure what the cheaper reference costs in false alarms.

No paper in the file reports any number for verification under architectural heterogeneity. We found no reported result on Lacuna for detection AUROC against cheating servers in HeteroFL or FjORD settings.

Why not just…?

Why not just retrain from scratch? Retraining is the reference, but it is what unlearning exists to avoid. Tarun et al. call verification-by-retraining an evaluation paradox. In federated settings it also requires every remaining client to come back online. Even the shortcut version, Rapid Retraining, costs 5–30% accuracy on remaining DomainNet domains (Towards Federated Domain Unlearning). Our audit needs no retrained model, and we report the audit's cost as a fraction of retraining.

Why not just add differential-privacy noise or prune the published delta? Both FUIA and DRAUN tested noise and pruning. FUIA reports accuracy collapsing from about 84% to about 63% under these defenses, and DRAUN describes a massive accuracy drop. Noise also corrupts the delta the verifier needs to read. We nonetheless include a noised-delta proof tuned to match our protocol's reconstruction SSIM, and report its detection AUROC and accuracy cost.

Why not just run IDI at full width on the client? This is the real threat to the proposal, and the full-width ablation settles it. If cheaters cannot hide information in core channels while cleaning the full-width features, the ladder is redundant. The case for the ladder rests on low-tier clients, whose influence lies entirely in the core slice, and on peripheral-only cheats. If full-width IDI catches these too, we will report the simpler method as the result.

Why not just use membership inference on outputs? Outputs can be silenced while features persist. The head distillation trap and activation-steering recovery of erased facts (Extracting Unlearned Information from LLMs with Activation Steering) show this directly. The baseline comparison measures how large this gap is in the federated setting.

Decisive Checks

The proposal succeeds if three things hold.

  1. On the low-tier departing client, the ladder detects head-only and peripheral-only cheats with clearly higher AUROC than the full-width ablation and the backdoor check.
  2. The adaptive cheater can only pass the ladder by spending server compute comparable to honest unlearning.
  3. The released statistic yields reconstructions at the level of random images, while published slice deltas reproduce DRAUN- and IGF-level fidelity.

The proposal fails if any of the following happen. The full-width ablation matches the ladder. Honest retraining trips the threshold more than rarely at small held-out sizes. A cheap cheater exists that is jointly suppressed across all widths. Each failure is informative. The first would mean features concentrate forgetting in shared ways across widths. The second would bound how small a client can be and still audit. The third would show that nested structure does not make faking expensive.