Send the Architecture, Prune the Weights: Leakage-Tiered Updates for Federated Hierarchical Search

Federated architecture search lets many devices jointly design a neural network without pooling their data. Each round, every device sends two kinds of numbers to a server. The first kind is a small set of architecture scores, which say which operations and connections the network should use. The second kind is a very large set of weight updates. Both kinds are a problem today. Weight updates are expensive to transmit, and a curious server can use them to reconstruct the device's private training images. The cheapest fix for both problems is to prune the updates aggressively, zeroing out all but the largest few percent of coordinates. That fix sits awkwardly with gradient-based architecture search, which picks between candidate operations using small, continuous differences in their scores and is already prone to collapsing onto trivial skip connections.

This note proposes to split the update along the network's own hierarchy. Hierarchical architecture search decomposes a network into a macro level (how stages connect) and a micro level (which operations sit inside each cell). Under this split, the architecture scores at both levels are tiny and hypothesized to leak little, so they are sent dense. Weight updates are pruned stage by stage, with the pruning rate set by how much each stage measurably leaks. The earliest stage is often never sent at all. The claim to test is simple: at a fixed upload budget, this tiered policy finds better architectures than uniform pruning, while giving reconstruction attacks no better results than uniform pruning does, including attacks that adapt their image prior.

Core Claim

Two published findings point in the same direction, and no one has combined them. Measuring and Controlling Split Layer Privacy Leakage Using Fisher Information introduces a per-layer leakage measure called diagonal Fisher information leakage (dFIL), whose inverse lower-bounds the mean squared error of any unbiased reconstruction. Using it, the paper shows that layers near the input leak far more than deep ones. Unchain the Search Space with Hierarchical Differentiable Architecture Search (H-DAS) shows that macro-level stage connections can be restricted to parameter-free operations, pooling and skips, and still reach 2.41% CIFAR-10 test error. Together these suggest that the signal architecture search needs is concentrated in a few hundred low-dimensional scores. The signal attackers need, and the bytes the network pays for, are concentrated in weight tensors, with leakage heaviest in the early stages.

Our claim is that matching compression strength to each hierarchy level does three things. It preserves the gradient signal that bi-level search depends on. Bi-level search means architecture scores are tuned on a validation split while weights are tuned on a training split. The tiered policy also cuts upload volume to the level of aggressive uniform pruning, and it holds reconstruction quality at or below uniform pruning. If uniform pruning already preserves search quality, the hierarchy-aware part is unnecessary, and we will report that result as such.

Research Program

The system starts from a federated version of the H-DAS search space. Each client holds a supernet, meaning a network that contains every candidate operation at once. The supernet has macro scores $\alpha^{mac}$ over stage-level connections and micro scores $\alpha^{mic}_s$ for the cell in each stage $s$. Following the decoupled design of Differentially-private Federated Neural Architecture Search, clients compute weight gradients on local training data and score gradients on a disjoint local validation split. That separation lets the privacy cost of the two channels be analyzed separately. Score gradients are sent dense in half precision.

Weight updates for stage $s$, which has $P_s$ parameters, keep only a fraction $k_s$ of coordinates by magnitude. Pruned coordinates are carried into the next round through error feedback, in the spirit of the shifted compression in ERIS. The server picks the keep fractions once per phase on public proxy data by solving

$$\min_{k_1,\dots,k_S} \sum_s k_s P_s \quad \text{subject to} \quad \mathrm{dFIL}_s(k_s) \le \tau \text{ for every stage } s,$$

Here $\mathrm{dFIL}_s(k_s)$ is the measured Fisher leakage of stage $s$'s pruned update about the input, and $\tau$ is a target leakage level. When no $k_s > 0$ meets the target, typically for the stem, that stage stays local and is personalized on each client.

The program has three parts. First comes a measurement study that does not yet exist. For each hierarchy level (macro scores, micro scores, and per-stage weights), we measure how well an attacker can reconstruct inputs from that level's gradients alone. Second, we characterize how bi-level search fails under compression. We run federated H-DAS with uniform top-k pruning at 90% and 99%, and with the median binarization of EdgeDetect. For each setting we record the fraction of skip connections in the derived cells and the stability of the score ranking across rounds. Third, we evaluate the tiered policy against these settings at matched bytes.

The hypothesis that architecture scores leak little is the one most likely to fail. Each score gradient is an inner product between an operation's output and the upstream gradient, and a strong image prior might extract identity from a few hundred such scalars. If that happens, the design changes: score gradients are clipped and noised. Clipping in a few hundred dimensions destroys far less useful signal than clipping in millions, so this fallback remains cheap.

Baselines

The naive reference point is federated H-DAS with dense, uncompressed updates, essentially FedNAS moved to the hierarchical space. FedNAS reports 81.24% test accuracy against FedAvg's 77.78% on a non-IID CIFAR-10 split. It defines the accuracy ceiling and the upload cost and leakage floor. The centralized ceiling for this search space is H-DAS's reported 2.41% CIFAR-10 test error at about 0.7 GPU-days.

The strongest competitor on this question is uniform magnitude pruning of the entire update, scores included. Data Reconstruction Attacks and Defenses: A Systematic Evaluation identifies gradient pruning as the best utility–privacy defense among those it compares, and it also delivers compression directly. It will run on the same clients, data partitions, search space, number of rounds and retraining recipe as our method. Its keep fraction will be tuned with the same budget of trials, and it will be compared at exactly matched upload bytes per round. No reported result was found on Lacuna for pruned federated architecture search, so this comparison is new.

A second strong baseline is DP-FNAS, which clips and noises updates for a flat cell supernet. Its reported figures are 3.4% test error with 8 parties against 2.8% centralized. We will rerun it on our splits rather than borrow those numbers. A third is EdgeDetect-style median binarization applied to the whole update. It reports reconstruction PSNR dropping from 31.7 dB to 15.1 dB, but on tabular intrusion data, so that figure is not directly comparable to images. A fourth is a FedorAS-style single-path sampler (Cross-device Federated Architecture Search), which cuts bytes by sending one sub-network per client.

Three ablations isolate the key ingredients. The first prunes scores and weights with the same keep fraction, testing whether dense scores matter. The second uses a single keep fraction for all stages but leaves scores dense, testing whether leakage-matched per-stage rates matter. The third keeps the stem local while uniformly pruning everything else.

Beating the strongest competitor means one of two outcomes at equal bytes. The tiered policy could give lower retrained test error at reconstruction PSNR no higher than uniform pruning's. Or it could give equal error with reconstruction PSNR at least 1 dB lower under the adaptive attack. One decibel is roughly the margin by which GI-NAS improved over the prior best attack, so it is a meaningful gap.

Why not just…?

Why not just prune everything uniformly? This is the real threat. Liu et al.'s systematic evaluation favors pruning precisely because it keeps utility, and continuous score gradients might survive it fine. Against this, H-DAS documents that differentiable search already drifts toward parameter-free skips unless a depth loss counteracts it. Pruning the small score gradients removes exactly the fine distinctions between competing operations. The second experiment settles the question directly. If skip-connection fraction and retrained error under 90% uniform pruning match the dense baseline, the hierarchy-aware channel is not needed, and the contribution reduces to the leakage measurements.

Why not just add differential privacy to a fixed network? Liu et al. find noise-based defenses give worse utility–privacy tradeoffs than pruning. GI-NAS shows that an attacker who searches over architectural priors defeats both additive noise and 90% pruning on fixed models. A fixed network also gives up the non-IID gains that FedNAS reports. We still include DP-FNAS so the cost of noise is measured on our splits rather than assumed.

Why not just freeze a pretrained backbone and send small adapters? Adapters cut bytes, but input-side prompts and adapters sit where dFIL says leakage is highest. They also assume every client can run a large backbone, which the device tiers in FedorAS contradict. This alternative is outside our search question, but we report a frozen-backbone variant with a stem-side adapter under the same attacks to quantify the leakage concern.

Why not just use secure aggregation? Encryption hides individual updates from the server. It does nothing against a participant who simply receives the decrypted global model, which FedInverse uses to reach 80% attack accuracy on MNIST. Running the search under cryptography, as in Towards Privacy-Preserving Neural Architecture Search, has only been shown feasible on MNIST. Secure aggregation is complementary to our method and can sit on top of it.

Proposed Experiments

The main benchmark is CIFAR-10 split across 8 and 32 clients by label-skew Dirichlet partitioning, using the H-DAS macro and micro space and its retraining protocol. CIFAR-100 under the same partitioning tests whether the conclusions hold with more classes. For the attack evaluation, we use the optimization-based inversion with cosine distance described in the file and the GI-NAS adaptive-prior attack, run on CIFAR-10 batches and on FFHQ-resolution face images in a smaller supernet. Each attack receives three inputs: the full update, score gradients alone, and each stage's pruned weight gradients alone. The metrics are retrained test error, bytes uploaded per round, total bytes to reach the dense baseline's accuracy, skip-connection fraction in the derived cells, and reconstruction PSNR and SSIM.

Three checks decide the outcome. The first asks whether reconstructions from score gradients alone stay near chance-level PSNR under the GI-NAS prior. If not, we switch to noised scores, and the low-leakage claim is withdrawn. The second asks whether uniform pruning at matched bytes measurably degrades the searched architecture relative to dense search. If not, the project's thesis is wrong. The third asks whether the tiered policy reaches the beating criterion above against uniform pruning under the adaptive attacker. That attacker matters because GI-NAS is the attack known to defeat pruning on fixed models.