Quaternion neural networks store each weight as a four-part number and multiply by a fixed rule, the Hamilton product. As a result they use a quarter of the parameters of an ordinary network of the same width. A striking finding is that they also survive aggressive pruning far better than ordinary networks: at 3% of the original parameters, a pruned quaternion 4-layer network beats a pruned real one on CIFAR-10 by more than ten points. That advantage fades in deeper models such as ResNet-18 and VGG-16, and nobody knows why. Separately, mathematicians have shown that a quaternion convolution can be split exactly, after a quaternion Fourier transform, into many small independent $2\times 2$ blocks. This makes its full set of singular values cheap to compute. Singular values measure how much a layer stretches its input along each direction. So far the split has been used for only one thing: finding the largest singular value and clipping it for stability.
This note proposes using the whole spectrum instead. During training, we apply a cheap penalty that pushes each deep quaternion convolution toward low rank at every frequency. We then compress the layer into a narrow quaternion bottleneck. The hypothesis is that deep quaternion networks lose their pruning advantage because ordinary training leaves their layers high-rank and poorly conditioned. If so, conditioning the full block spectrum should restore the graceful degradation that shallow quaternion networks show. The same experiments also test whether the quaternion structure contributes anything beyond what real-valued spectral compression already achieves.
Core Claim
The claim has two parts, and both can be checked.
The first part is a mechanism. The quaternion pruning advantage in deep networks is limited by how the singular values are spread across frequency blocks, not by depth as such. A penalty on the sum of per-frequency singular values, which is a nuclear-norm surrogate for rank, applied through the exact block decomposition, should let deep quaternion ResNets keep accuracy at compression levels where both magnitude-pruned quaternion networks and SVD-compressed real networks collapse.
The second part is about structure. When a quaternion matrix is truncated in its own quaternion spectral domain, its singular values come in structured pairs. Truncating whole pairs yields an operator that is still exactly quaternionic. The compressed network is therefore still a quaternion network, with its weight-sharing bias intact. A real-valued SVD of the same layer, cut without respecting that pairing, gives no such guarantee.
Relation to Prior Work
Two papers supply the method.
Unlocking the matrix form of the Quaternion Fourier Transform and Quaternion Convolution derives the matrix form of the quaternion Fourier transform. It shows that the square norm of a quaternion convolution kernel becomes block-diagonal with $2\times 2$ blocks, and it uses this to clip the largest singular value. That clipping cuts cost from $O(N^6)$ to $O(N^2\log N)$ and raises QResNet32-small on CIFAR-10 from 81.43% to 85.80%. The paper keeps only $\sigma_{\max}$ and treats single-kernel blocks. We use every block and extend the construction to layers with many input and output channels.
Block-Diagonalization of Quaternion Circulant Matrices with Applications proves the general $1\times1$/$2\times2$ block-diagonalization and uses low-rank truncation in that domain, but only to compress stored color video, never network weights during training.
Two papers motivate the proposal.
Neural Networks at a Fraction with Pruned Quaternions found the sparsity crossover and its fading in deep networks. It used only unstructured magnitude pruning and offered no mechanism for the fading.
Low-Rank Prehab shows that standard training leaves real weights in high-rank regions. It also shows that a conditioning phase with a nuclear-norm or stable-rank surrogate rescues SVD compression: ViT-B/16 at 60% compression goes from 28.38% to 48.12%. Our proposal is essentially the quaternion, per-frequency analogue of Prehab. Whether the quaternion part matters is exactly what the ablations below test.
Integrating Independent Layer-Wise Rank Selection with Low-Rank SVD Training is the strongest real-valued compression competitor. It trains the SVD factors directly with Hoyer regularization.
PHNNs and Quaternion Approximate Networks set the bar for small hypercomplex models built directly at a reduced size. They constrain what counts as a win, because a compressed model must beat simply training a small hypercomplex model from scratch.
Research Program
The multi-channel block spectrum. A quaternion convolution with $C_{in}$ input and $C_{out}$ output quaternion channels uses circular padding, which makes the convolution exactly circulant. The first step extends the Sfikas construction to this case. Applying the permuted quaternion Fourier transform along one axis $\mu$ turns the layer into, at each frequency pair, a complex matrix $B_f$ of size $2C_{out}\times 2C_{in}$. Its singular values are those of the full convolution operator. This is derived analytically and checked numerically against brute-force SVD of the real-expanded operator on small layers, with the same $<10^{-11}$ agreement target the original paper reports.
Standard ResNets use zero padding, where the circulant form is only approximate. We therefore train circular-padded variants and measure how far the penalty computed on the circulant form drifts from the true spectrum under zero padding.
The penalty. The conditioning objective adds to the task loss
$$\lambda \sum_{\ell}\sum_{f} \frac{\|B^{(\ell)}_f\|_*}{\|B^{(\ell)}_f\|_F},$$
where $\ell$ indexes layers, $f$ indexes frequency blocks, $\|\cdot\|_*$ is the sum of singular values, and $\|\cdot\|_F$ is the Frobenius norm. Dividing by the Frobenius norm makes the penalty scale-free, so it rewards concentrating energy in few directions rather than simply shrinking the weights.
This penalty matches the compression step for a concrete reason. Factoring a convolution into a $k\times k$ quaternion convolution into $r$ channels, followed by a $1\times1$ quaternion convolution back, bounds each $B_f$ to rank $2r$. A low per-frequency rank is therefore necessary for the factorization to be accurate. It is not sufficient, because different frequencies could use different subspaces. We measure the alignment of the top singular subspaces across frequencies and, if needed, add a term coupling the frequency blocks to a shared basis.
Compression. After conditioning, each layer is replaced by the quaternion bottleneck. Its initialization comes from a frequency-averaged truncated quaternion SVD, cut on whole pairs so the result stays quaternionic. Per-layer ranks are chosen by a global energy threshold on the block spectra. A short recovery fine-tune follows.
Diagnosis of depth degradation. On magnitude-pruned deep quaternion networks, we record per-layer stable rank and $\sigma_{\max}$ from the block spectra. We then test whether the layers where accuracy collapses are the ones with inflated rank or norm.
Baselines
The naive reference. Two naive points anchor the comparison. One is the dense network, both the quaternion ResNet-18 and QResNet32-small from the Sfikas repository, and their real counterparts. The other is a quaternion network trained from scratch at the compressed width, which is the "just build it small" option. PHNN models with larger $n$, which cut parameters to $1/n$, are included at matched parameter counts as the other from-scratch hypercomplex reference.
Magnitude pruning. The first existing-method baseline is iterative global magnitude pruning of quaternion and real networks, as in Neural Networks at a Fraction with Pruned Quaternions. Its reported number is that quaternion Conv-4 beats real Conv-4 on CIFAR-10 by more than 10% at 3% remaining parameters. We rerun it on the deep models, where the paper says the advantage fades.
Strongest real-valued competitor. The strongest competitor is low-rank SVD training with layer-wise rank selection from Integrating Independent Layer-Wise Rank Selection with Low-Rank SVD Training. It reports 0.892 CIFAR-10 test accuracy at a compression ratio of 0.155 on ResNet-20. We also include real-valued Prehab conditioning followed by SVD, from Low-Rank Prehab, applied to real ResNets. No CIFAR number is reported for Prehab on Lacuna; its reported figure is ViT-B/16 at 48.12% under 60% compression.
Fairness and the winning criterion. All methods get the same data, augmentation, epoch budget including conditioning and recovery epochs, and the same hyperparameter-search budget over the strength and rank-threshold settings. Beating the strongest baseline means one of two things. Either higher CIFAR-10 accuracy than 0.892 at a compression ratio of 0.155 or below on a ResNet-20-scale model, reproduced in our setup. Or a matched-accuracy advantage that grows with compression across the whole accuracy–compression curve.
Ablations that isolate the key ingredient. Three ablations separate out what the proposal adds. The first applies Lipschitz clipping of $\sigma_{\max}$ alone before magnitude pruning, testing whether norm control rather than rank control explains any recovery. The second applies the same nuclear-norm penalty computed on the real-expanded operator through an ordinary FFT, with $4C\times4C$ blocks and truncation that ignores quaternion pairing. This isolates the quaternion spectral formulation. The third skips conditioning and applies direct block-spectral truncation, testing whether conditioning is needed at all.
Why not just…?
Why not just compress an ordinary real network with SVD or Prehab? This is the default option, and it may win at moderate compression, where real networks are stronger. The evidence against it lies at extreme compression. Real networks fail sharply once pruning removes about 90% of their parameters, and quaternion networks start with a 75% structural reduction before any pruning. The head-to-head accuracy–compression curves against Low-Rank SVD Training and Prehab will settle whether this advantage holds in deep models. If real Prehab matches us everywhere, the proposal fails.
Why not just run the real-valued per-frequency SVD on the expanded quaternion layer? This is a genuine threat. The singular values of the real-expanded operator are those of the quaternion operator, repeated. A real FFT-domain penalty is therefore tractable and could give the same accuracy. What the quaternion form adds is threefold: blocks four times smaller in each dimension, truncation that is guaranteed to stay quaternionic, and the ability to choose the axis $\mu$. The second ablation measures exactly this difference. If the two match in accuracy, we will report the contribution as computational and diagnostic rather than as an accuracy gain.
Why not just train a smaller quaternion or PHNN model from scratch? PHNNs already matches ResNet-50 on ImageNet with 66% fewer parameters. Quaternion Approximate Networks reaches 95.12% on CIFAR-10 with 717K parameters. So a small model built directly is strong. Over-parameterization research suggests that training large and then compressing finds better solutions than training small, but that must be shown here. The matched-parameter from-scratch baselines are therefore decisive. If they match conditioned compression, the practical case collapses to the diagnostic finding alone.
Why not just clip the Lipschitz constant, which already helps? Clipping controls only the largest stretch a layer can apply, whereas compression requires the layer's energy to be concentrated in few directions. The first ablation tests whether clipping alone before pruning restores the deep-network quaternion advantage. If it does, the mechanism is norm explosion rather than rank inflation, and the full-spectrum penalty is unnecessary.
Proposed Experiments
The main experiments use CIFAR-10 and CIFAR-100 with Conv-4, QResNet32-small, and quaternion ResNet-18, together with real counterparts matched in width and in parameter count. Each method produces an accuracy-versus-remaining-parameters curve from 50% down to 1%. We also evaluate on the replicated CIFAR-10 test set from Do CIFAR-10 Classifiers Generalize to CIFAR-10?, which reports 4–10% drops for ordinary models, to check whether compressed quaternion networks are also more robust to this natural shift. An ImageNet ResNet-18 run is a stretch goal.
Three results are decisive. The first is whether conditioned spectral compression restores, in ResNet-18, a quaternion-over-real gap at 3% of parameters comparable to the more-than-10% gap reported for Conv-4. The second is whether the per-layer stable rank of magnitude-pruned deep quaternion networks predicts where accuracy collapses. The third is whether the quaternion-domain penalty beats its real-expanded counterpart at equal compression, or merely matches it at lower cost. We also report the wall-clock cost of the penalty against brute-force spectrum computation, and how sensitive the method is to the axis $\mu$, which the original paper left uncharacterized for learned weights.
References
- Unlocking the matrix form of the Quaternion Fourier Transform and Quaternion Convolution: Properties, connections, and application to Lipschitz constant bounding
- Block-Diagonalization of Quaternion Circulant Matrices with Applications
- Neural Networks at a Fraction with Pruned Quaternions
- Low-Rank Prehab: Preparing Neural Networks for SVD Compression
- Integrating Independent Layer-Wise Rank Selection with Low-Rank SVD Training for Model Compression: A Theory-Driven Approach
- PHNNs: Lightweight Neural Networks via Parameterized Hypercomplex Convolutions
- Quaternion Approximate Networks for Enhanced Image Classification and Oriented Object Detection
- Do CIFAR-10 Classifiers Generalize to CIFAR-10?