Training an image classifier with differential privacy (a guarantee that the trained model reveals almost nothing about any single training image) works by clipping each example's gradient and adding Gaussian noise to every trainable weight. The noise is added per coordinate, so it grows with the number of weights, while the useful signal does not. The most effective fixes project the noisy gradient onto a small subspace that contains most of the true signal. Finding that subspace usually requires public data from the same domain, and that data is often missing in exactly the settings where privacy matters, such as medical images. Quaternion networks are layers that treat each RGB pixel as a single four-part number and multiply with Hamilton's rule. They tie every four real weights into one shared weight, which fixes a specific subspace in advance. That subspace is not arbitrary: it encodes the assumption that color channels move together, which is a known property of natural images.
This note proposes to test one claim. The quaternion constraint acts as a data-independent gradient projection, and it improves private training only to the extent that the true gradient actually lies inside the Hamilton-tied subspace. If the claim holds, a color-structured architecture could replace public-data subspace estimation. It would also make a measurable, falsifiable prediction about when this works and when it fails.
Core Claim
Consider a real-valued layer of the same width as a quaternion layer. Let $g$ be its per-example gradient, of dimension $p$. Let $P$ be the orthogonal projection onto the $p/4$-dimensional subspace of weights obeying the Hamilton sign-and-sharing pattern. Projecting onto this subspace amounts to averaging the four signed entries that the quaternion layer ties together. We define the capture ratio
$$\rho = \frac{\mathbb{E}\,\|P g\|^2}{\mathbb{E}\,\|g\|^2},$$
which is the fraction of gradient energy that survives the tying.
At a fixed clip norm and noise multiplier, the full layer receives noise energy proportional to $p$. The tied layer receives noise energy proportional to $p/4$ and keeps a fraction $\rho$ of the signal. The tied layer therefore has the better per-step signal-to-noise ratio roughly when $4\rho > 1$. This is a one-step heuristic that ignores how clipping and optimization dynamics interact.
The threshold $1/4$ is exactly what a random subspace of that dimension captures on average. This explains why random projections performed poorly in Bypassing the Ambient Dimension, which projects noisy gradients onto a subspace estimated from 100–150 public samples. It also sharpens the question: does Hamilton structure beat $1/4$ on real color data, in which layers, and by how much?
The framing also resolves a tension in the literature. Unlocking High-Accuracy DP Image Classification through Scale shows that large real-valued networks train well under privacy. Adapter and tensor-train work finds that shrinking the trainable parameter set helps (DP Adapters, TTLoRA-DP). Both findings are consistent if the raw parameter count matters less than whether the trainable subspace is aligned with the gradient. The same logic implies that the quaternion $1/4$ parameter reduction described in Demystifying the Hypercomplex is not a benefit by itself.
Relation to Prior Work
Two strands of prior work supply the method. The first is the construction of quaternion convolutions: Quaternion CNNs couple channels through rotations around the gray axis. Demystifying the Hypercomplex introduces Kronecker-parameterized hypercomplex layers (PHNNs) that learn the sharing pattern instead of fixing it. The second strand is the subspace view of private SGD, from Bypassing the Ambient Dimension and Subspace Differential Privacy.
The closest intersecting work is Deep Quaternion Features for Privacy Protection. It hides intermediate features by applying a secret 3D rotation and running rotation-equivariant layers in the cloud. It concerns inference only, has no formal guarantee, and does not use input color structure. We study private training with a standard $(\epsilon,\delta)$ guarantee, and the quaternion structure enters only as an architectural prior. Quaternion-Hadamard Network addresses robustness to adversarial test inputs, which is unrelated to training privacy.
Two works constrain the design. Quaternion CNNs reports that the benefit shrinks on low-saturation imagery. The same paper, along with the quaternion VAE work, reports that quaternion operations run about twice as slow as real ones. That slowdown hurts more under private training, where per-example gradients are already expensive.
Research Program
We will first implement quaternion and PHNN convolutions as real operations with shared parameters. This makes them compatible with standard per-example clipping. The privacy accounting is then unchanged: clipping and noise apply to the free parameters, and the tied real weights are a deterministic function of those parameters.
Next, we will measure the capture ratio $\rho$ layer by layer. The measurement uses a matched-width real network trained non-privately, taking gradients at checkpoints along its training trajectory. We expect $\rho$ to be highest in the first layer, where "channels" really are R, G, B. We expect it to fall toward $1/4$ in deep layers, where feature groups have no physical meaning. This motivates a hybrid design with quaternion early layers and a real-valued remainder, whose split point is chosen from $\rho$ measured on non-private proxy data rather than tuned on the private set.
Third, we will train privately on CIFAR-10 from scratch and on DermaMNIST-C. DermaMNIST-C is the lesion-ID-partitioned correction from Investigating the Quality of DermaMNIST and Fitzpatrick17k, which found that about 13.47% of lesions leaked across partitions in the original split. It is a realistic setting with no public in-domain data. Oxford Flowers serves as a high-chroma fine-grained case. Privacy budgets are $\epsilon \in \{1, 2, 4, 8\}$ with $\delta = 10^{-5}$.
Fourth, we will test whether the per-layer $\rho$ measured beforehand predicts the private accuracy gap across layers, datasets, and color manipulations. This is the scientific payoff: a cheap diagnostic that tells a practitioner whether color-structured tying will help before any privacy budget is spent.
Baselines
The naive reference is a real-valued Wide-ResNet of the same width trained with the recipe of Unlocking High-Accuracy DP Image Classification through Scale: group normalization, weight standardization, large batches and augmentation multiplicity. The strongest reported result in the file is 81.4% CIFAR-10 accuracy at $(8, 10^{-5})$ for a 40-layer Wide-ResNet without extra data (De et al.). Our compute budget will likely not reproduce that exact figure. We will therefore run this recipe ourselves at a fixed batch size and step count, and apply the identical recipe, tuning grid and seeds to every other arm.
The most dangerous baseline is a parameter-matched real network: a narrower Wide-ResNet, or a grouped-convolution network, with the same number of free parameters as the quaternion model. It tests whether any fourfold reduction helps. We will pair it with the key ablation, a random-tying network. That network uses the same four-way sharing pattern as Hamilton multiplication but with randomly drawn signs and positions, frozen at initialization. It has exactly the same dimension and speed as the quaternion model, so any advantage of Hamilton over random tying isolates the color prior.
A learned PHNN is included as an alternative. Its sharing pattern is learned non-privately on a small public proxy set, or learned under the same private budget, and it shows whether a fixed algebra is needed at all.
The strongest subspace competitor is PDP-SGD from Bypassing the Ambient Dimension. We will give it 150 public images: in-distribution for CIFAR-10, and out-of-domain natural images for DermaMNIST-C, which reflects its realistic use there. The strongest post-processing competitor is FFTKF from Spectral and Temporal Denoising for DP Optimization, which filters high-frequency noise from gradients. It reports 73.73% on CIFAR-10 at $\epsilon=4$ with a Wide-ResNet-40, against 72.13% for DiSK and 54.50% for DP-Adam. FFTKF is complementary to our approach, so we also run it on top of the hybrid quaternion model.
Beating the baselines means two things. Hamilton tying must beat both the parameter-matched and random-tying networks by more than seed variance at $\epsilon \le 4$. It must also match or exceed PDP-SGD on DermaMNIST-C without using public data.
Why not just…?
Why not just use a narrower real network? If parameter count were all that mattered, a quarter-size real network would match the quaternion model at half the run time, and the proposal would reduce to "use fewer weights." The scale results of De et al. already argue against pure parameter reduction. Whether the quaternion prior beats simple shrinkage has not been tested, and the parameter-matched and random-tying arms settle it directly. If they tie with the quaternion model, we will report that the effect is dimensional rather than chromatic, and the claim fails.
Why not just estimate the subspace from public data with PDP-SGD? PDP-SGD needs in-domain public data and a clear gap in the gradient spectrum, and Bypassing the Ambient Dimension notes that projection fails without them. On CIFAR-10 with in-distribution public samples, PDP-SGD may well win; that is a real threat, and we will report it plainly. The regime where our approach should matter is DermaMNIST-C with only out-of-domain public data. There the comparison shows whether a physics-derived subspace is more reliable than an estimated one.
Why not just filter the noise afterwards with FFTKF? FFTKF treats the gradient as a flat vector and removes high-frequency parameter-space noise. It does not change which directions receive noise, so the two approaches should stack rather than compete. The combined arm tests this. If FFTKF alone erases the gap, our subspace adds nothing beyond what it already removes.
Why not just scale up? Scale works well when pre-training data and very large batches are available. The 86.7% ImageNet result of De et al. uses a pre-trained NFNet-F3, and De et al. note that private training is about nine times slower than standard training. Private medical-image settings usually lack both the data and the compute. Our comparisons are therefore made at fixed compute, and we will report the wall-clock cost of the twofold quaternion slowdown alongside accuracy.
Proposed Experiments and Decisive Checks
The central check is a color-rotation intervention. We apply a fixed random orthogonal 3×3 color transform to every CIFAR-10 image. This preserves all information but misaligns the gray axis around which quaternion kernels rotate. The theory predicts that the first-layer $\rho$ drops and the Hamilton-over-random advantage shrinks, while the real baselines are unaffected.
A second intervention replicates grayscale across the three channels, the low-saturation extreme reported in Quaternion CNNs. There, $\rho$ changes in a predictable direction, and the private-accuracy gap should follow it.
Across all layers, datasets and interventions, we will regress the private accuracy gap on the pre-measured $\rho$. We will report accuracy against $\epsilon$ curves and a membership-inference AUC audit following the protocol style of TTLoRA-DP.
The proposal is falsified if Hamilton tying does not beat random tying at matched dimension, or if $\rho$ fails to predict where the gains appear.
References
- Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification
- Unlocking High-Accuracy Differentially Private Image Classification through Scale
- Differentially Private Adapters for Parameter Efficient Acoustic Modeling
- Privacy Enhanced PEFT: Tensor Train Decomposition Improves Privacy Utility Tradeoffs under DP-SGD
- Demystifying the Hypercomplex: Inductive biases in hypercomplex deep learning [Hypercomplex Signal and Image Processing]
- Quaternion Convolutional Neural Networks
- Subspace Differential Privacy
- Deep Quaternion Features for Privacy Protection
- Quaternion-Hadamard Network: A Novel Defense Against Adversarial Attacks with a New Dataset
- Investigating the Quality of DermaMNIST and Fitzpatrick17k Dermatological Image Datasets
- Spectral and Temporal Denoising for Differentially Private Optimization