The Computational Cost of Complex Valued Neural Networks

110 concepts91 papers10 proposals

In fields like signal processing, magnetic resonance imaging, and quantum-inspired language modeling, data is inherently wave-like. To capture this, researchers often turn to Complex-Valued Neural Networks (CVNNs), which represent information using both magnitude and phase. While these architectures offer superior mathematical expressivity—allowing for more robust face recognition under lighting changes (Yang et al.) and better reconstruction of speech signals (Lee et al.)—they come with a significant "computational tax."

Understanding why these models are expensive requires looking past the number of parameters to the underlying arithmetic, software implementation, and hardware mismatch.

The Arithmetic Tax

The most fundamental source of overhead is the transition from real to complex arithmetic. In a standard neural network, a single multiplication is a one-to-one operation. However, as Zhang et al. observe, a single complex multiplication $(a+bi)(c+di)$ requires four real-valued multiplications and two additions:

  1. $ac$
  2. $bd$
  3. $ad$
  4. $bc$ Resulting in $(ac - bd) + (ad + bc)i$.

This 4x multiplication overhead means that even if a complex-valued layer has the same number of logical connections as a real-valued one, the number of floating-point operations (FLOPs) is significantly higher. Gu et al. note that this increased demand poses a major challenge for deploying high-performance models on resource-constrained or low-power hardware.

The Implementation Gap

Beyond raw math, CVNNs suffer from an "engineering overhead" because modern deep learning ecosystems were built for real numbers. Saragadam et al. point out that most frameworks are primarily optimized for real-valued operations, making the integration of complex arithmetic less straightforward.

This leads to two specific efficiency bottlenecks:

  1. Autograd Fragmentation: As Oh et al. explain, frameworks typically implement complex operations by tracking real and imaginary parts as separate nodes in a computational graph. This fragmentation increases the complexity of the automatic differentiation (autograd) process, leading to higher memory overhead and slower training times.
  2. Kernel Mismatch: Standard GPU kernels are optimized for real-valued Fused Multiply-Add (FMA) operations. Arjovsky et al. and Jing et al. argue that the lack of native, optimized support for complex-valued operations in these libraries necessitates custom implementations, which often fail to reach the peak hardware performance of their real-valued counterparts.

The Parameter-Efficiency Paradox

A common argument for using hypercomplex representations, such as quaternions, is that they reduce the total number of parameters. For instance, Parcollet and Ravanelli demonstrate that Quaternion LSTMs can use up to 3.3x fewer parameters than standard RNNs by leveraging the Hamilton product to capture internal structural dependencies.

However, this parameter efficiency does not always translate to speed. Chaudhary and Fang et al. note that the Hamilton product and other structured products introduce significant computational intensity per parameter. In practice, a model with fewer parameters might actually require more Multiply-Accumulate (MAC) operations, leading to a training slowdown unless specialized GPU kernels are developed (Parcollet and Ravanelli).

The Deployment Barrier

The cumulative effect of these overheads is a barrier to real-time deployment. In a systematic study of monaural speech enhancement, Wu et al. concluded that complex-valued networks consistently demand more computational resources (MACs) than real-valued networks to achieve similar performance levels.

For edge devices, this cost is often prohibitive. Kong et al. and Fu et al. emphasize that without significant model compression or hardware-specific acceleration, heavy complex-valued architectures remain unsuitable for mobile or resource-constrained platforms. Even when using techniques like sparsification, Nazarov and Burnaev warn that storage-based compression does not automatically result in proportional execution speedups because the underlying arithmetic complexity remains unchanged.

Key Foundational Work

Recognize Color Face Images Using Complex Eigenfaces (2006) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_complex_valued_computational_overhead_51e42c77,
  title = {complex-valued computational overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/complex-valued-computational-overhead/rdr_51e42c77d7154b3b8ff48f5093528bec},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_complex_valued_computational_overhead_51e42c77_v1,
  title = {complex-valued computational overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/complex-valued-computational-overhead/rdr_51e42c77d7154b3b8ff48f5093528bec/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake