In fields like signal processing, magnetic resonance imaging, and quantum-inspired language modeling, data is inherently wave-like. To capture this, researchers often turn to Complex-Valued Neural Networks (CVNNs), which represent information using both magnitude and phase. While these architectures offer superior mathematical expressivity—allowing for more robust face recognition under lighting changes (Yang et al.) and better reconstruction of speech signals (Lee et al.)—they come with a significant "computational tax."
Understanding why these models are expensive requires looking past the number of parameters to the underlying arithmetic, software implementation, and hardware mismatch.
The Arithmetic Tax
The most fundamental source of overhead is the transition from real to complex arithmetic. In a standard neural network, a single multiplication is a one-to-one operation. However, as Zhang et al. observe, a single complex multiplication $(a+bi)(c+di)$ requires four real-valued multiplications and two additions:
- $ac$
- $bd$
- $ad$
- $bc$ Resulting in $(ac - bd) + (ad + bc)i$.
This 4x multiplication overhead means that even if a complex-valued layer has the same number of logical connections as a real-valued one, the number of floating-point operations (FLOPs) is significantly higher. Gu et al. note that this increased demand poses a major challenge for deploying high-performance models on resource-constrained or low-power hardware.
The Implementation Gap
Beyond raw math, CVNNs suffer from an "engineering overhead" because modern deep learning ecosystems were built for real numbers. Saragadam et al. point out that most frameworks are primarily optimized for real-valued operations, making the integration of complex arithmetic less straightforward.
This leads to two specific efficiency bottlenecks:
- Autograd Fragmentation: As Oh et al. explain, frameworks typically implement complex operations by tracking real and imaginary parts as separate nodes in a computational graph. This fragmentation increases the complexity of the automatic differentiation (autograd) process, leading to higher memory overhead and slower training times.
- Kernel Mismatch: Standard GPU kernels are optimized for real-valued Fused Multiply-Add (FMA) operations. Arjovsky et al. and Jing et al. argue that the lack of native, optimized support for complex-valued operations in these libraries necessitates custom implementations, which often fail to reach the peak hardware performance of their real-valued counterparts.
The Parameter-Efficiency Paradox
A common argument for using hypercomplex representations, such as quaternions, is that they reduce the total number of parameters. For instance, Parcollet and Ravanelli demonstrate that Quaternion LSTMs can use up to 3.3x fewer parameters than standard RNNs by leveraging the Hamilton product to capture internal structural dependencies.
However, this parameter efficiency does not always translate to speed. Chaudhary and Fang et al. note that the Hamilton product and other structured products introduce significant computational intensity per parameter. In practice, a model with fewer parameters might actually require more Multiply-Accumulate (MAC) operations, leading to a training slowdown unless specialized GPU kernels are developed (Parcollet and Ravanelli).
The Deployment Barrier
The cumulative effect of these overheads is a barrier to real-time deployment. In a systematic study of monaural speech enhancement, Wu et al. concluded that complex-valued networks consistently demand more computational resources (MACs) than real-valued networks to achieve similar performance levels.
For edge devices, this cost is often prohibitive. Kong et al. and Fu et al. emphasize that without significant model compression or hardware-specific acceleration, heavy complex-valued architectures remain unsuitable for mobile or resource-constrained platforms. Even when using techniques like sparsification, Nazarov and Burnaev warn that storage-based compression does not automatically result in proportional execution speedups because the underlying arithmetic complexity remains unchanged.
Key Foundational Work
Recognize Color Face Images Using Complex Eigenfaces (2006) — contributed 1 concept elements to this direction.