Hospitals that want to train a shared model without pooling patient records often use ADMM, the alternating direction method of multipliers. In ADMM, each site solves its own small problem, a coordinator averages the results, and a penalty weight ρ pulls the sites toward agreement. Two findings motivate this proposal. First, ADMM's speed depends heavily on ρ: a poorly chosen value can slow convergence by orders of magnitude or stop it altogether. Second, the automatic methods that choose ρ estimate the shape of the loss surface from differences between successive iterates. Under differential privacy (DP), where each site deliberately adds calibrated random noise to what it sends so that no single patient's record can be inferred, those differences are mostly noise. Current private ADMM systems handle this by fixing ρ and tuning it by hand. In a clinical setting, every tuning run spends privacy budget.
We propose a penalty-adaptation rule that the coordinator runs using only the noisy messages it already receives. It therefore costs no additional privacy. It also subtracts the known noise contribution from its curvature estimates, which the coordinator can do because it knows the noise variance each site used. Because clinical sites differ in how much noise they must add, the rule assigns each site its own penalty. The question we want to answer is whether a coordinator can choose ρ as well as an oracle grid search, at zero extra privacy cost, across a network of hospitals with unequal privacy requirements.
Core Claim
The claim has three parts.
- Adaptation is free in privacy terms. Adapting ρ from privatized messages is post-processing, so it spends no privacy budget.
- Naive adaptation fails under DP noise. Applying an existing spectral rule unchanged to noisy iterates gives curvature estimates biased toward zero. Its safeguard, which updates ρ only when a local linear fit is good, then almost never fires, so the rule quietly falls back to a fixed ρ.
- A debiased rule works. Correcting the estimates with the known noise variances, and allowing a separate penalty per site, recovers most of the iteration savings seen in non-private adaptive ADMM. The same correction keeps noisy low-compliance hospitals from destabilizing the consensus.
The second part is falsifiable and would be a useful negative result by itself.
Relation to Prior Work
The optimization side supplies the method and the motivation.
- Why ρ matters. Ghadimi et al. show that on quadratic problems the convergence rate collapses away from a well-balanced ρ. Poon et al. show that on non-smooth problems ADMM spirals toward the optimum, and a bad ρ widens the spiral.
- Spectral adaptation, the rule we modify. Adaptive ADMM with Spectral Penalty Parameter Selection (AADMM) estimates local curvature from Barzilai–Borwein ratios. These are ratios of inner products between successive changes in the variables. AADMM sets ρ from their geometric mean and updates only when a correlation safeguard passes. It converged on all 23 datasets tested, where fixed-ρ ADMM failed on 13. Its estimates assume clean iterate differences, and that assumption is the one we remove.
- Sharpness-based adaptation. ADMM without a Fixed Penalty Parameter doubles ρ in stages tied to a local sharpness exponent that is unknown in practice. We use it as a comparison, not a building block.
The privacy side supplies the setting and the gap.
- Private ADMM with a fixed ρ. DP-ADMM adds decaying Gaussian noise to linearized primal updates. Differentially Private Federated Learning via Inexact ADMM perturbs local objectives instead. VIMADMM applies ADMM to vertically split data and reports that its method is "not overly sensitive to ρ within a reasonable range." All three hold ρ fixed.
- Direct evidence of sensitivity. FDR-SVM states that its ADMM solver "was found to be sensitive to its hyperparameters, requiring more careful tuning" than subgradient methods.
- Unequal noise across sites. Inclusive, Differentially Private Federated Learning for Clinical Data scales each clinic's noise by a compliance score. Including the noisier clinics still improved accuracy by 1% to 15%. That work uses gradient-averaging aggregators and never considers per-site penalties. Our proposal takes its unequal-noise regime as the test condition.
- The analogous coupling. Decoupling Regularization and Privacy in DP Ridge Regression shows that one knob controlling both statistical bias and privacy sensitivity is a bottleneck. It also notes that hand cross-validation wastes privacy budget. In ADMM, ρ couples consensus speed with the sensitivity of the local update, which we explain below.
Research Program
Consider consensus ADMM over $m$ sites. Site $i$ releases a noisy local solution $\tilde x_i^k = x_i^k + \xi_i^k$, where $\xi_i^k \sim \mathcal{N}(0, \sigma_{i,k}^2 I_d)$. The noise scale $\sigma_{i,k}$ is set by the privacy mechanism and is public. The coordinator updates the consensus variable and the dual variables $\lambda_i$ from these released vectors alone.
AADMM's curvature estimates are built from inner products of changes such as $\Delta \tilde x_i = \tilde x_i^{k} - \tilde x_i^{k-t}$. The noise terms at the two iterations are independent, so the expected squared norm is biased upward by a known amount:
$$\mathbb{E}\,\|\Delta \tilde x_i\|^2 = \|\Delta x_i\|^2 + d\,(\sigma_{i,k}^2 + \sigma_{i,k-t}^2).$$
We subtract this term, and the analogous cross-term corrections, which are also computable because the dual updates are linear in $\tilde x_i$. We then average over a short window of rounds to reduce variance. The debiased quantities feed both the geometric-mean rule and the correlation safeguard. When the debiased denominator is not clearly positive, the rule holds ρ fixed.
The rule is applied per site, producing penalties $\rho_i$. We expect the adapted penalties to settle higher for noisier sites, since a stiffer pull toward consensus damps their oscillations. The experiments test this prediction rather than assume it.
The analysis addresses the coupling of ρ with privacy. For a linearized local step with gradient clipping at norm $C$, the step's sensitivity scales as $C/(n_i(\rho_i+\eta))$, where $n_i$ is the site's sample count and $\eta$ is a fixed proximal weight. Raising $\rho_i$ therefore also lowers the noise the site needs for a given per-round privacy cost. The accountant handles this cleanly because each $\rho_i$ is a post-processed, public quantity. It does create a feedback loop, though: the rule could inflate ρ simply because doing so shrinks the noise, even when consensus suffers. We will run a variant that calibrates noise to a frozen reference ρ, which separates this effect from genuine curvature adaptation.
On the theory side, we aim to bound the bias and variance of the debiased Barzilai–Borwein estimate as a function of the ratio $d\sigma^2/\|\Delta x\|^2$. We will also prove convergence of the safeguarded scheme, using a bounded, eventually-frozen ρ in the style of standard adaptive-ADMM analyses. Rates beyond that are not promised.
Baselines
Fixed ρ with a default value (do-nothing reference). This is DP-ADMM as published (DP-ADMM), run with ρ = 1 and identical clipping, noise schedule, and accounting. It shows what practitioners get without tuning.
Oracle fixed ρ. We sweep a logarithmic grid of ρ values, and select using the test objective with no privacy charged for tuning. This is the strongest competitor, because any adaptive rule is trying to match a well-chosen constant for free. We also report a privacy-honest version that splits the budget across grid points. Beating the oracle is not required. Matching it within a small accuracy margin and using no tuning budget would count as a win, and beating the privacy-honest version would count as a clear win.
Naive AADMM on noisy iterates. This is AADMM with its published safeguard threshold ($\epsilon^{cor}\approx 0.2$), applied unchanged to the released messages. It is the ablation that isolates debiasing. We will report how often its safeguard fires as noise increases.
Global versus per-site penalties. Our debiased rule with a single global ρ is the ablation that isolates per-site penalties.
Residual balancing and staged doubling. Residual balancing on noisy residuals, and staged doubling of ρ from LA-ADMM, are the two other standard adaptive rules.
Non-ADMM reference. DP-FedAdam, which Inclusive DP FL found more robust than FedAvg to non-uniform noise, run under the same privacy budget.
Fairness conditions for all comparisons. Every method gets the same privacy parameters (ε, δ), the same number of communication rounds, the same clipping norm, and the same number of seeds. Any method that needs tuning gets the same grid size, with the tuning cost either charged or explicitly flagged as uncharged.
Why not just…?
Why not just grid-search a fixed ρ on public or synthetic data? This is the most serious threat. If a single ρ transfers across hospitals and noise levels, our method adds little. The existing evidence against it comes from non-private settings: static ρ failed on 13 of 23 datasets in AADMM, and calibration breaks across data scales in Zhou et al.. None of that says what happens under DP noise. We will tune ρ on public MedMNIST splits and transfer it to private clinic partitions and to MIMIC-III. If the transferred ρ reaches within 1 point of our method's accuracy across noise levels, we will report that as the finding.
Why not just run AADMM as is, since it is already post-processing? It is post-processing, so its privacy cost is zero. Our hypothesis is that noise inflates the squared-norm terms in its estimates and suppresses its correlation safeguard, so it silently reverts to fixed ρ. The naive-AADMM ablation, together with the logged safeguard firing rate as a function of $d\sigma^2/\|\Delta x\|^2$, tests this directly. If naive AADMM works, the debiasing contribution is void, and the note becomes an empirical study of adaptive ADMM under DP.
Why not just drop ADMM for DP-FedAvg or FedAdam? In the files we have, ADMM's advantage under DP is communication. VIMADMM needed roughly 43 times less communication than its gradient-based baseline, VAFL, to reach 65.0% accuracy on CIFAR, and fewer rounds means less privacy spent. ADMM also handles non-smooth and constrained objectives, such as the distributionally robust formulation in FDR-SVM. The DP-FedAdam baseline will show whether the advantage survives once ρ is chosen well.
Why not just use residual balancing? AADMM shows that residual balancing is not scale-invariant: rescaling the units of a problem changes its behaviour. Its residuals are also noisy under DP. We will include residual balancing with and without feature rescaling.
Proposed Experiments
Convex problems with known optima. We use L1-regularized logistic regression and hinge-loss SVM. The non-smooth objectives are chosen deliberately, because that is where the spiral behaviour appears. The datasets are:
- Adult, for comparability with DP-ADMM. That paper reports runtime only; no reported accuracy was found on Lacuna for DP-ADMM on Adult.
- MIMIC-III in-hospital mortality, using MIMIC-Extract features, partitioned by care unit to mimic separate sites (MIMIC-III).
- PneumoniaMNIST and BreastMNIST from MedMNIST v2, using frozen features from a publicly pretrained network. We reproduce the 16-clinic, compliance-scaled noise design of Inclusive DP FL, which reports 1% to 15% accuracy gains for inclusion but no per-dataset absolute accuracy that we found.
Deep-learning extension. We run an inexact-ADMM small CNN on FEMNIST. The target to beat is 72.40% test error at $\bar\epsilon = 0.05$ for DP-IADMM-Trust, reported in DP FL via Inexact ADMM.
Metrics. We measure:
- rounds needed to reach a fixed suboptimality at fixed privacy cost;
- final test AUC or accuracy at ε ∈ {0.5, 1, 4};
- sensitivity to the initial ρ₀ over $10^{-4}$ to $10^{4}$;
- accuracy on the noisiest clinics' own validation data.
Decisive checks.
- The debiased rule should hold its round savings flat as noise rises, while naive AADMM's savings collapse toward fixed-ρ behaviour.
- Final accuracy should be nearly invariant to ρ₀ across eight orders of magnitude, where fixed ρ is not.
- In the inclusive 16-clinic setting, per-site penalties should beat a global ρ, and the gain should concentrate where compliance scores are most spread out.
- The frozen-reference-ρ variant should show how much of the gain is curvature adaptation rather than noise reduction from inflating ρ.
If the transferred public-data ρ matches us on all of these, we conclude that fixed tuning suffices in this regime.
References
- Optimal Parameter Selection for the Alternating Direction Method of Multipliers (ADMM): Quadratic Problems
- Trajectory of Alternating Direction Method of Multipliers and Adaptive Acceleration
- Adaptive ADMM with Spectral Penalty Parameter Selection
- ADMM without a Fixed Penalty Parameter: Faster Convergence with New Adaptive Penalization
- DP-ADMM: ADMM-based Distributed Learning with Differential Privacy
- Differentially Private Federated Learning via Inexact ADMM
- Improving Privacy-Preserving Vertical Federated Learning by Efficient Communication with ADMM
- FDR-SVM: A Federated Distributionally Robust Support Vector Machine via a Mixture of Wasserstein Balls Ambiguity Set
- Inclusive, Differentially Private Federated Learning for Clinical Data
- Decoupling Regularization and Privacy in Differentially Private Ridge Regression and ERM
- Fast Hybrid Algorithm for Big Matrix Recovery
- MIMIC-Extract: a data extraction, preprocessing, and representation pipeline for MIMIC-III
- MIMIC-III, a freely accessible critical care database
- MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification