TL;DR
This paper introduces the Parameterized Hypercomplex Convolutional (PHC) layer, a method that constructs convolutional weight tensors as a sum of learnable Kronecker products. This design lets networks learn algebraic interaction rules directly from data for any user-chosen dimension $n$, slashing free parameters to approximately $1/n$ without requiring rigid hypercomplex definitions (like quaternions) or dummy zero-padding for 3D data like RGB images.
What Problem Does This Paper Address?
Scaling up convolutional neural networks typically leads to massive parameter counts. While hypercomplex neural networks—such as Quaternion Neural Networks (QNNs)—exploit algebraic symmetries (like the Hamilton product) to reduce parameter counts by $4\times$, they suffer from major structural limitations:
- Rigid Dimensionality: Standard quaternion algebra strictly operates in 4D. For 3D data such as standard RGB images, models must append an artificial zero-channel, introducing unnecessary computation and dummy values.
- Poor Scalability to Higher Dimensions: Extending hypercomplex algebra to higher dimensions (e.g., octonions in 8D or sedenions in 16D, useful for multi-microphone audio with phase) requires manually defining non-commutative and non-associative algebras, which are mathematically awkward and rarely implemented in standard deep learning pipelines.
- Parameterization gap in convolutions: While Parameterized Hypercomplex Multiplication (PHM) had generalized linear layers using Kronecker products, convolutional layers—the backbone of computer vision and audio modeling—lacked a corresponding formulation.
Methods
The authors introduce Parameterized Hypercomplex Convolution (PHC) layers:
-
Kronecker Sum Formulation: The convolutional weight tensor $H \in \mathbb{R}^{s \times d \times k \times k}$ (with input channels $s$, output channels $d$, and kernel size $k$) is constructed as: $$H = \sum_{i=1}^n A_i \otimes F_i$$ where each $A_i \in \mathbb{R}^{n \times n}$ represents the learnable algebra rules, and $F_i \in \mathbb{R}^{\frac{s}{n} \times \frac{d}{n} \times k \times k}$ holds the convolutional filter bases.
-
Learnable Domain Rules: Instead of fixing $A_i$ to rigid Clifford or quaternion multiplication tables, $A_i$ and $F_i$ are trained end-to-end via standard backpropagation.
- Arbitrary Dimensionality $n$: The user can specify any integer $n$:
- $n=3$ natively processes RGB channels without padding zeros, saving $\approx 66\%$ of parameters.
- $n=4, 8, 16$ accommodates multichannel Ambisonic audio signals with magnitude and phase.
- Degrees of Freedom: The parameter count scales as $\mathcal{O}(n^3 + \frac{sdk^2}{n}) \approx \mathcal{O}(\frac{sdk^2}{n})$ when $sdk^2 \gg n^3$, reducing parameters to roughly $1/n$ compared to standard convolutions.
Datasets
The authors evaluated their approach across image classification and 3D sound event detection:
- Image Classification: SVHN, CIFAR-10, CIFAR-100, and large-scale ImageNet (ILSVRC).
- Sound Event Detection (SED): L3DAS21 Challenge Task 2 dataset (15 hours of simulated 3D office Ambisonics audio recordings using 1 to 2 first-order Ambisonics microphones with 14 sound classes).
Key Insights and Ideas
- RGB Images in Their Natural Domain ($n=3$): Unlike QNNs that must pad RGB images into 4D space, PHC with $n=3$ models cross-channel correlations natively, outperforming both real baselines and $n=4$ models in multiple image tasks.
- Parameter Sharing as Regularization: Reducing parameters by $1/n$ through structured Kronecker sharing often improves test accuracy rather than degrading it, because it enforces inter-channel feature reuse. The authors verified that simply shrinking a standard CNN's width by 75% degrades accuracy, proving the performance boost stems from the hypercomplex inductive bias, not smaller capacity.
- Data-Driven Algebra Discovery: Toy experiments show that PHC layers can automatically rediscover the Hamilton product rule when trained on synthetic quaternion convolutions.
- Practical Efficiency Gains: In audio tasks (SEDnets), PHC models achieved equal FLOPs to baselines but delivered measurable wall-clock speedups in training and inference time while cutting GPU checkpoint memory by up to 74%.
Key Results
- CIFAR-100 (ResNet-152): PHC ($n=4$) scored 66.44% accuracy vs. 62.05% for real-valued ResNet-152 and 61.27% for Quaternion ResNet-152—a $>4\%$ accuracy gain despite using 75% fewer parameters (13.4M vs. 52.6M).
- ImageNet (ResNet-50): PHC ($n=3$) reached 68.58% top-1 accuracy vs. 67.99% for baseline ResNet-50, using 66% fewer parameters (9.6M vs. 25.7M).
- Sound Event Detection (L3DAS21):
- 1 Microphone (4 channels): PHC SEDnet ($n=2$) achieved an $\text{F}_{\text{score}}$ of 0.680 and $\text{SED}_{\text{score}}$ of 0.355, beating the real baseline (0.637 / 0.406) and quaternion baseline (0.580 / 0.468).
- 2 Microphones + Phase (16 channels): PHC SEDnet scaled up to $n=16$ (sedenion domain), matching the baseline while reducing convolutional parameters by 94% (0.1M vs. 1.6M).
Limitations and Future Work
- Channel Divisibility Constraints: Input and output channel dimensions must be divisible by $n$, requiring minor adjustments to standard layer widths when experimenting with arbitrary values of $n$.
- Higher FLOP Overhead: While parameter counts drop by $1/n$, FLOP counts remain virtually identical or slightly higher due to the reconstruction of the full kernel via Kronecker summation.
- Future Directions: The authors highlight combining PHC layers with neural network pruning to further push the boundaries of lightweight edge deployment.
Summary updated