A growing body of work argues that the shape of a neural network decides how well it survives private training. Private training here means differentially private stochastic gradient descent (DP-SGD), which clips each example's gradient and adds Gaussian noise so that no single training record can be inferred. The reported findings are concrete. Activations that keep negative values beat Tanh. Residual networks shrug off clipping where AlexNet does not. Vision Transformers leak more than CNNs. Each of these claims rests on a handful of runs, in one software stack, on one kind of hardware. Separate work on reproducibility shows that the seed, the GPU and the library version can move a metric by as much as a published improvement. We propose to measure, in a controlled factorial experiment, how much of each architectural privacy claim is caused by the architecture and how much by the execution stack around it.
The main hypothesis is specific. For DP training, the part of the "environment" that matters most is probably not the GPU. It is the set of hidden privacy-relevant defaults in the training library: whether batches are drawn by Poisson sampling or by shuffling, how per-example clipping is implemented, and how noise is sampled. These defaults change the actual privacy guarantee. They may also interact with architecture, because clipping's effect depends on how gradients are distributed. If this is right, architecture rankings can flip when only library defaults change, while the nominal privacy budget $\epsilon$ stays the same. The project's outputs are a containerized harness that treats execution environment and DP implementation as explicit experimental factors, together with a variance decomposition of five published architectural claims.
Core Claim
Published comparisons treat the architecture as the only variable and everything else as fixed. We claim that the claimed effects should instead be reported relative to three other sources of variation:
- Seed (arbitrary variation): random initialization and data order.
- Hardware and software stack (incidental variation): GPU, CUDA, framework version.
- DP implementation choices: sampling scheme, clipping code path, noise sampler.
Some claims will survive this test and some will not. We predict that the gaps the literature reports as large will survive. These include DPNAS's 15–20% margin over NASNet and DARTS under identical DP constraints. We predict that the small or interaction-driven gaps will be fragile. These include the clipping comparison on EMNIST and the claimed leakage ordering between Transformers and CNNs under DP.
What is new is not containerization. It is using containers to vary the environment deliberately as a factor, and decomposing privacy-architecture effects against that variation.
Grounding in Prior Work
The architectural claims we test come from four papers.
- DPNAS searches architectures under DP-SGD. It finds SELU, Group Normalization and max pooling favorable, and reports 68.33% CIFAR-10 accuracy at $(\epsilon,\delta)=(3,10^{-5})$.
- Understanding Clipping for Federated Learning ties clipping damage to how tightly updates concentrate. On non-IID EMNIST, clipping costs ResNet-18 0.10% accuracy and AlexNet 3.60%.
- How Does a Deep Learning Model Architecture Impact Its Privacy? finds Swin Transformers more exposed than ResNets to membership inference (guessing whether a record was in the training set) and to gradient inversion. - Leave-one-out Distinguishability predicts, through infinite-width Gaussian-process analysis, that ReLU leaks less than GeLU in 95% of tested cases.
These papers supply the hypotheses under test. None of them varies the execution stack.
Three papers motivate the concern that the stack matters.
- Exploring Variation of Results from Different Experimental Conditions separates methodological, arbitrary and incidental variation. It shows that seed changes alone moved LSTM BLEU from 84.47 to 89.59, and that a 2017 environment had become unbuildable.
- Instrumentation bias documents the "wrapper trap": hard-coded defaults in high-level wrappers that silently change what the underlying code computes.
- How Private are DP-SGD Implementations? shows the most important such default for DP. Libraries batch by shuffling while accountants assume Poisson sampling. In the setting it studies ($\sigma=0.5$, $T=10{,}000$ steps), this turns a reported $\epsilon\approx 2$–$3$ into a lower bound above 10.
That paper works at the level of batch queries and does not ask whether architecture interacts with the batching mode. That interaction is the question we add.
Methods are borrowed from three sources.
- From Bencher we take the design that wraps conflicting dependency stacks in separate Docker or Singularity images behind a common interface. This lets one experiment driver dispatch the same training job to stacks with different versions.
- From SciAgent we take append-only provenance logs. We use them so that every run records its sampling mode, clipping path and library hashes.
- From Quantified Reproducibility Assessment we take the small-sample-corrected coefficient of variation $CV^*$ as the stability statistic.
The closest bridging work is Epistemic Parity. It asks whether scientific conclusions survive DP synthesis, and finds that modeling structure matters more than $\epsilon$. Our question is the mirror image: whether conclusions about DP architectures survive a change of execution stack. That paper studies tabular synthesizers, not DP-SGD architectures, and it does not vary the runtime. DP-Auditorium supplies the divergence estimators we use to put empirical lower bounds on privacy loss. It audits mechanisms, not architecture comparisons across stacks.
Research Program
We first build a harness: a set of container images that differ along controlled axes.
- Framework and DP-library versions: several PyTorch and Opacus releases spanning a few years.
- Deterministic-kernel flags: on versus off.
- Hardware: two GPU generations plus a CPU-only image.
A single driver dispatches identical job specifications to each image. Separately from the image, each job sets three DP implementation switches:
- Poisson versus shuffled batching.
- Per-example clipping through the library's default path versus a reference implementation we write.
- Floating-point Gaussian noise versus a discrete Gaussian sampler.
Every run writes its full configuration, image digest and per-step batch sizes to an append-only log. The last of these lets us confirm afterward which sampling scheme actually executed.
We then rerun five published comparisons in a full factorial design:
- DPNAS-style SELU, Group Normalization and max pooling versus a Tanh CNN.
- The same network with ReLU versus SELU.
- ResNet-18 versus AlexNet under client-level clipping on EMNIST.
- ResNet versus a small Swin Transformer, comparing membership-inference exposure under DP.
- ReLU versus GeLU, comparing empirical leakage at finite width.
For each outcome $y$ (test accuracy, membership-inference AUC, or an empirical $\epsilon$ lower bound from canary audits), we fit a random-effects model:
$$ y = \mu + a_{\text{arch}} + e_{\text{env}} + s_{\text{impl}} + (a{\times}s)_{\text{arch,impl}} + (a{\times}e)_{\text{arch,env}} + \varepsilon_{\text{seed}}. $$
The terms are:
- $\mu$: the grand mean.
- $a$: the architecture effect.
- $e$: the container or hardware effect.
- $s$: the DP implementation effect.
- The two product terms: interactions between architecture and implementation, and between architecture and environment.
- $\varepsilon_{\text{seed}}$: residual variation across seeds.
Two summary statistics follow. The first is the share of variance attributable to each term. The second is a robustness ratio $R = |\Delta_{\text{arch}}| / \sigma_{\text{non-arch}}$, where $\Delta_{\text{arch}}$ is the mean architecture gap and $\sigma_{\text{non-arch}}$ is the standard deviation contributed by all non-architecture terms combined. We also report the probability that the ranking flips across the factor cells, and $CV^*$ for each architecture.
Baselines
The naive reference point is the published single-configuration comparison: one seed or a few, one environment, library defaults. Concretely, these are the DPNAS figure of 68.33% on CIFAR-10 at $\epsilon=3$, the 0.10% versus 3.60% clipping drops on EMNIST from Zhang et al., and the qualitative Transformer-versus-CNN leakage ordering from Zhang et al. The file reports no numerical membership-inference AUC for that last comparison, so no reported result was found on Lacuna for the exact attack scores. We reproduce each published configuration first, in an image matching its original stack as closely as we can, so that later deviations are measured against a faithful rerun rather than a re-implementation.
The strongest existing practice, and the strongest competitor, is a seed-replicated comparison in one pinned environment with library defaults. It is cheaper than our design and is what a careful practitioner already does. We run it with the same architectures, data, privacy budget, training epochs and hyperparameter search budget as the full factorial: each architecture gets the same number of tuning trials, and those trials are run in the reference image only. Our design beats it if the factorial reveals ranking flips or interaction variance that the seed-only design reports as zero. Concretely, the test is a non-architecture variance share that the seed-only design underestimates by more than its own confidence interval, or at least one published claim whose $R$ falls below 2 only once environment and implementation are varied.
Two ablations isolate the key ingredient. The first varies hardware and framework version while holding the DP implementation switches fixed. The second varies only the switches inside one image. If the hypothesis is correct, the second ablation accounts for most of the non-seed variance and the first accounts for little.
Why not just…?
Why not just run more seeds in one pinned environment? This is the real threat, and the experiment is built to settle it. If the environment and implementation terms in the model come out negligible, then seed replication is sufficient. We would report that as a clean result telling the field to stop worrying about containers for DP architecture work. We do not expect it, because seed replication cannot change the batching mode. How Private are DP-SGD Implementations? shows that the batching mode alone changes the true privacy level by a large factor. Whether it also changes architecture rankings is exactly the interaction a seed-only design cannot see.
Why not just pin dependencies with a lockfile instead of containers? Lockfiles fix Python packages but not CUDA, system libraries or kernels. Exploring Variation reports a baseline that became unbuildable as low-level libraries aged. Instrumentation bias reports operating-system-dependent outputs from identical code. Containers here are only a means of making the environment a controlled factor, so if the hardware-only ablation shows no effect, we will say lockfiles suffice for this purpose.
Why not just audit the final checkpoints with DP-Auditorium or membership-inference attacks? We do use such audits as outcome measures. Used post hoc, however, they produce a leakage number without saying which implementation choice caused it. Without logged sampling modes and per-step batch sizes, a shuffle-induced gap would be misattributed to architecture. The provenance logs exist to prevent that misattribution.
Why not just use the infinite-width analysis to predict leakage without training? The Leave-one-out Distinguishability analysis is fast, but it assumes infinite width and ideal arithmetic. It does not model clipping, finite-precision noise sampling (shown exploitable, 92.56% canary detection on MNIST with Opacus, in Are We There Yet?) or batching mode. We treat its ReLU-versus-GeLU prediction as a hypothesis and test whether the finite-width ordering holds across the factorial.
Proposed Experiments and Decisive Checks
The image benchmarks are MNIST, FashionMNIST and CIFAR-10, trained centrally with DP-SGD at $(\epsilon,\delta)=(3,10^{-5})$ to match DPNAS. Federated clipping is tested on EMNIST and Shakespeare with client-level DP, following Zhang et al. Membership-inference evaluation is run on CIFAR-10 and CIFAR-100 for the ResNet-versus-Swin comparison, at a small enough scale to fit inside the full design.
The design per benchmark is 5 architecture pairs, 4 container images, 4 implementation configurations and 5 seeds. At the small-CNN scale DPNAS uses, this is a few hundred private runs, which is feasible.
The outcome measures are:
- Accuracy.
- $CV^*$.
- Variance shares from the random-effects model.
- Ranking-flip probability.
- Membership-inference AUC.
- Empirical $\epsilon$ lower bounds from canary insertion, using DP-Auditorium's estimators.
Three results would be decisive.
- Interaction check. We will test whether the architecture-by-implementation interaction is significant for the clipping comparison. If ResNet-18 versus AlexNet clipping damage on EMNIST changes order or magnitude between Poisson and shuffled batching, then the concentration argument of Zhang et al. depends on an implementation detail its accounting assumes away.
- Large-gap check. The DPNAS-versus-Tanh gap should keep $R>2$ in every cell. Failure here would mean even the strongest architectural claim in the literature is not robust.
- Leakage-ordering check. We will test whether the claim that Transformers leak more, measured as an empirical privacy lower bound rather than raw attack AUC, persists when both models run under real Poisson sampling. If the ordering survives only under shuffling, architecture-level leakage claims under DP need to be re-run with matched implementations.
Each check has a stated failure mode. Together they determine whether "architectural privacy sensitivity" is a property of networks or partly a property of the machines and libraries that train them.
References
- DPNAS: Neural Architecture Search for Deep Learning with Differential Privacy
- Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacy
- How Does a Deep Learning Model Architecture Impact Its Privacy? A Comprehensive Study of Privacy Attacks on CNNs and Transformers
- Leave-one-out Distinguishability in Machine Learning
- Exploring Variation of Results from Different Experimental Conditions
- Instrumentation bias in the use and evaluation of scientific software: recommendations for reproducible practices in the computational sciences
- How Private are DP-SGD Implementations?
- Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization
- Audit-Grade Harness for Agent-driven Scientific Computation Workflows
- Quantified Reproducibility Assessment of NLP Results
- Epistemic Parity: Reproducibility as an Evaluation Metric for Differential Privacy
- DP-Auditorium: A Large-Scale Library for Auditing Differential Privacy
- Are We There Yet? Timing and Floating-Point Attacks on Differential Privacy Systems