Several groups of organizations, such as hospitals or banks, may want to fine-tune one shared language model on data they cannot pool. The cheapest known way to coordinate them is to have each party send only the sign (+1 or −1) of its proposed change to every weight, then move each weight in whichever direction most parties voted for. Recent sign-based optimizers report communication cuts of 16× to 960× this way. But the coordinator sees every party's individual votes, and no one has measured how much private training data those votes reveal. Homomorphic encryption, which lets a server add or multiply numbers it cannot read, could hide the votes. In practice it has been too slow and too bulky for anything beyond small models or a final classification layer.
This note proposes a specific way out. A majority vote over $K$ parties is the sign of a small bounded integer, so it can be computed exactly under encryption with a shallow polynomial, without the approximate activations or expensive "bootstrapping" refreshes that make encrypted deep learning slow. The protocol then reveals only the majority direction, which every party receives as the model update anyway. It hides the individual votes and the vote counts. The research question is whether that guarantee is worth its cost. That requires measuring two things side by side: how much each level of disclosure (raw votes, vote counts, majority only) helps a data-reconstruction attacker, and what each level costs in bandwidth and latency at language-model scale.
Core Claim
Sign-majority aggregation is unusually well suited to homomorphic encryption. Each party contributes a value in $\{-1,0,+1\}$, so the encrypted sum $s$ per coordinate is an integer with $|s|\le K$. The output we want, $\mathrm{sign}(s)$, is then a function on at most $2K+1$ known points. In an exact integer scheme such as BFV (a lattice-based scheme over a prime plaintext modulus $t>2K$), it can be written as an odd interpolating polynomial
$$\mathrm{sign}(s) = s \cdot g(s^2) \pmod t,$$
where $g$ has degree about $K/2$. The multiplicative depth is therefore roughly $\log_2 K + 2$. For cross-silo settings with up to a few dozen parties, the circuit stays within a leveled scheme's budget with no bootstrapping. This polynomial is exact, not approximate, and it maps a tie ($s=0$, for example after a party drops out) to 0. That matches the ternary $\{-1,0,1\}$ update that BitDP already finds works for linear-layer gradients.
The contrast with prior encrypted learning is sharp. SecureNets and HETAL approximate nonlinearities over continuous ranges, and HETAL needs domain-extension polynomials up to $[-128,128]$ because softmax inputs grow during training. Our nonlinearity has a tiny discrete domain fixed by the number of parties.
The second claim is about what to encrypt. The votes should be computed on short local weight deltas, meaning the change in a party's weights over $H$ local steps, rather than on single-step gradients. This borrows the delta-compression observation from BitDelta that the sign of a fine-tuning delta carries most of its value. It also borrows the reduced-frequency synchronization idea of DiLoCo. Summing $H$ steps averages out stochastic noise, which should raise the probability that parties agree on each sign. That matters because ABSignSGD identifies agreement above 0.5 as the condition for majority vote to work at all. Fewer rounds also means fewer encrypted rounds to pay for.
Research Program
The work has three parts, and each can fail on its own.
Part 1: leakage measurement. We measure how much each disclosure level leaks. The motivating conflict comes from the literature itself. Mixed Quantization Enabled Federated Learning reports that 8-bit quantization collapses DLG reconstruction on CIFAR-10 from 55.3 to 8.1 dB PSNR. That suggests coarse signals are safe. However, that defense relies on withholding the dequantization scales, so it is not a test of quantization alone. On the other side, Gradient Inversion with Generative Image Prior reconstructs images when 99% of gradient entries are zeroed, by searching a generator's latent space. That suggests coarse signals still leak once the attacker has a good prior.
No paper in the file tests 1-bit signs against prior-backed attacks. We will adapt three attacks to observe sign vectors or vote counts instead of real-valued gradients, replacing the matching loss with a sign-agreement or count-agreement loss:
- Inverting Gradients and the generative-prior attack for images.
- LAMP for text, following the setup in Evaluating Selective Encryption Against Gradient Inversion Attacks.
- Cross-round tracking in the style of Beyond Inferring Class Representatives. Repeated sign streams from one party may reveal its data distribution even when no single round does.
Part 2: protocol construction. We build two protocol tiers.
- Tier A encrypts additively and reveals the vote counts to the key holders. It packs many small integers per ciphertext, in the manner of BatchCrypt, using about $\log_2(2K+1)$ bits per coordinate.
- Tier B evaluates the exact sign polynomial on SIMD-packed BFV ciphertexts (one coordinate per slot). It then uses threshold decryption held jointly by the parties, so that only the majority vector is ever decrypted.
Ciphertext expansion is the main expected obstacle. A Systematic Comparison of Encrypted ML Solutions identifies bandwidth as the bottleneck, and one-coordinate-per-slot packing is far less dense than Tier A's packing. Two structural facts from the sign-optimizer literature reduce the volume. Block-coordinate updates in ABSignSGD touch one block of layers per round, so only that block's votes need encrypting. The embedding and output-head layers that BitDelta leaves uncompressed can stay frozen or be handled separately. We will report measured bytes per parameter per round. We are not projecting them in advance.
Part 3: end-to-end training and the final artifact. We fine-tune with the protocol and check that the accumulated sign updates, scaled per block as in ABSignSGD and BAS, give accuracy comparable to plaintext sign voting. The final fine-tune is itself a scaled sum of sign deltas, so it can be stored and served as a BitDelta-style compressed delta. We will check that this round trip costs little accuracy.
Threat model. The server is honest-but-curious, and the parties are semi-honest key holders. We will also run the active "fishing model" label attack of Breaking Secure Aggregation as a stress test. That attack recovered labels with 100% accuracy through secure aggregation by manipulating a normalization layer. We do not claim Tier B defeats it. We will report whether reducing the output to the majority direction degrades it.
Baselines
Naive references. The no-privacy reference is plaintext synchronous fine-tuning with full-precision gradients averaged across parties. It sets the utility ceiling and the worst-case leakage. The no-encryption sign reference is plaintext majority voting as in ABSignSGD-MV, which the authors report cuts communication 960× versus PyTorch DDP on Qwen3-8B. It is the strongest communication baseline: any encrypted tier must be judged by how much of that saving it gives back. Both run on the same models, data splits, number of parties, and step budget as our method, with learning rates tuned on the same grid.
Strongest encrypted aggregation baseline. This is BatchCrypt: 8-to-16-bit quantized gradients packed and summed under additive encryption, with analytical clipping. Its authors report 23× to 93× training speedups and 66× to 101× communication reduction over unbatched encryption with under 1% accuracy loss, on AlexNet over CIFAR-10 with 9 clients. We will reimplement it on our transformer workloads at the same key security level and party count. To beat it, Tier B must reach comparable task accuracy while revealing strictly less, namely the majority vector instead of the full sum, at wall-clock cost per round within a small constant factor of BatchCrypt.
Prior binarized encrypted aggregation. EdgeDetect combines binarized gradients with additively homomorphic Paillier encryption. It reports inversion PSNR degraded from 31.7 to 15.1 dB on a tabular intrusion-detection dataset with shallow networks. It is the closest prior system but reveals the decrypted sum. We include it on CIFAR-10 to anchor our leakage numbers against a published point.
Selective encryption. Following Gu et al., this baseline encrypts only the top 10–40% of gradient entries by magnitude. That paper reports protection at LPIPS ≥ 0.3 and ROUGE-1 ≤ 0.2 against IG and LAMP at batch size 1. We run it at their protection thresholds on the same text and image tasks.
Differential privacy. We add Gaussian noise to deltas before voting, tuned to match Tier B's measured reconstruction failure, and report the resulting utility.
Ablations. Tier A versus Tier B isolates the value of hiding vote counts. Per-step gradient signs versus $H$-step local-delta signs isolates the delta-compression ingredient, measured by sign-agreement probability, rounds to target loss, and leakage per round.
Why not just…?
Why not just send plaintext signs, since 1-bit signals may already leak nothing? The file contains evidence in both directions. Quantization alone appeared to defeat DLG on LeNet, but that defense hid its scales. Generative-prior attacks survive 99% sparsification. If our Part 1 measurements show that prior-backed IG and LAMP attacks on raw sign vectors stay at chance on CIFAR-10 and SST-2, then encryption is unnecessary for honest-but-curious servers. The note would then report that, along with the cross-round tracking results, which could still change the answer.
Why not just use secure aggregation masking or Tier A, which reveal only the sum and are much cheaper? This is the most serious threat to the proposal. Pairwise-mask secure aggregation gives the same disclosure as Tier A at far lower cost than lattice ciphertexts. The file reports no measured overhead for it, so we will implement it as a reference. Tier B is only justified if vote counts measurably help an attacker. That is plausible when $K$ is small: with two parties, the count reveals the other party's sign outright. It is less plausible for large $K$. The Tier A versus Tier B leakage comparison, run across $K$, decides this directly. If counts add nothing beyond the majority, we will recommend masking-based aggregation and say so.
Why not just add differential privacy noise instead of encrypting? In the file, matching quantization's protection with DP dropped MNIST accuracy to 78–85%, against 96.67% for the quantized scheme. Sign voting is also known to be fragile at low agreement probability, and noise lowers agreement further. Our DP baseline, tuned to equal leakage, measures this utility cost directly on language-model fine-tuning.
Why not just use LoRA to shrink what needs encrypting? Fewer trainable parameters would cut ciphertext volume. But BitDelta reports that full fine-tuning deltas are high-rank, and ABSignSGD reports a 6% accuracy gain over memory-efficient baselines, including LoRA, on OpenMathInstruct-2. We will run LoRA under Tier B as an additional configuration. If it matches block-sign full fine-tuning at lower cost, it becomes the recommended deployment.
Proposed Experiments
Leakage experiments. We use CIFAR-10 and ImageNet 64×64 with ResNet and ViT models, attacked by IG and a GAN-prior attack. We use CoLA, SST-2, and Rotten Tomatoes with BERT and GPT-2, attacked by LAMP. Metrics are PSNR, LPIPS, and ROUGE-1, reported at batch sizes 1 through 32 for each disclosure level: full gradient, sign vector, count, and majority. We also run LIA-SA label recovery on CIFAR-10/100.
Utility and cost experiments. We fine-tune Qwen-scale models on OpenMathInstruct-2 and evaluate on GSM8K and MT-Bench, with 4, 8, and 16 simulated parties over Ethernet-class links. We report:
- accuracy;
- sign-agreement probability;
- bytes per parameter per round;
- wall-clock time per round;
- BitDelta-compressed size of the final delta.
Decisive checks. Three outcomes decide the proposal.
- Encryption is warranted at all only if raw sign vectors permit reconstruction clearly above chance under prior-backed attacks.
- Tier B's main design choice survives only if majority-only disclosure brings reconstruction to the selective-encryption protection thresholds while vote counts do not.
- The method is viable only if Tier B's per-round cost, with block-wise updates, stays close enough to BatchCrypt's to keep most of the communication savings of plaintext sign voting at matched GSM8K accuracy.
If the first check fails, the contribution becomes the negative leakage result. If the second fails, the recommendation is masked summation.
References
- BitDP: Ultra-low-bit Communication for Data Parallelism in LLM Training
- SecureNets: Secure Inference of Deep Neural Networks on an Untrusted Cloud
- HETAL: Efficient Privacy-preserving Transfer Learning with Homomorphic Encryption
- BitDelta: Your Fine-Tune May Only Be Worth One Bit
- Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
- Arbitrary-Order Block SignSGD for Memory-Efficient LLM Fine-Tuning
- Mixed Quantization Enabled Federated Learning to Tackle Gradient Inversion Attacks
- Gradient Inversion with Generative Image Prior
- Evaluating Selective Encryption Against Gradient Inversion Attacks
- Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning
- BatchCrypt: Efficient Homomorphic Encryption for Cross-Silo Federated Learning
- A Systematic Comparison of Encrypted Machine Learning Solutions for Image Classification
- Breaking Secure Aggregation: Label Leakage from Aggregated Gradients in Federated Learning
- EdgeDetect: Importance-Aware Gradient Compression with Homomorphic Aggregation for Federated Intrusion Detection