Phase Preserving Non-linearities in Complex Neural Networks

This direction is merged into another direction: complex-valued neural networks. Its last version stays readable and citable here.

Version 1 of 134 papers46 conceptsfrozen 2026-10-04

In standard deep learning, we treat data as a collection of magnitudes. If a feature is important, its value is large; if it is irrelevant, it is zeroed out. However, many physical signals—from the echoes of a seismic blast to the interference patterns in an MRI—are fundamentally wave-like. These signals are best described by complex numbers, where information is split between magnitude (how strong the signal is) and phase (where the wave is in its cycle).

The challenge in building Complex-Valued Neural Networks (CVNNs) is that our most successful tool, the non-linear activation function, was designed for real numbers. If you apply a standard ReLU to a complex number, you risk "breaking" the phase, effectively throwing away the structural information that makes complex representations useful in the first place.

The Phase-Magnitude Dilemma

In a complex-valued neuron, the output is a point on the complex plane. When we apply a non-linearity, we have two primary choices: do we treat the real and imaginary parts as independent channels, or do we treat the complex number as a single geometric entity?

As Chiheb Trabelsi and colleagues detailed in the foundational Deep Complex Networks, the simplest approach is the Split-ReLU. This applies a standard ReLU to the real and imaginary components independently. Geometrically, this partitions the complex plane into four quadrants, potentially nullifying the signal if it falls into the "negative" regions of the plane. While computationally efficient and surprisingly effective for tasks like audio classification, Split-ReLU is "phase-agnostic"—it can drastically alter the angle of the complex vector.

Preserving Structure via Magnitude-Only Gating

For tasks where the phase encodes critical information—such as the relative timing in speech or the orientation of edges in an image—researchers often prefer magnitude-based activations.

The intuition, first explored in early work like Phasor Neural Networks by André J. Noest, is to decouple the "intensity" of a neuron from its "content." By applying a non-linearity (like a sigmoid or a threshold) only to the magnitude $|z|$ while leaving the phase $\theta$ untouched, the network can scale a signal's importance without distorting its timing or geometric orientation.

This mechanism is the "secret sauce" for several specialized architectures:

  • Equivariant Networks: In Harmonic Networks, Daniel E. Worrall and his team use magnitude-only non-linearities to ensure that if an input image rotates, the network's internal representations rotate perfectly in sync (rotation equivariance).
  • Recurrent Stability: Amar Shah and Yoshua Bengio introduced Unitary Evolution Recurrent Neural Networks to solve the vanishing gradient problem. By using unitary matrices (which preserve the norm of a vector) and magnitude-based activations, they ensure that gradients neither explode nor vanish over thousands of time steps.

Why Phase Matters: From MRI to Heart Sounds

Why go through the mathematical trouble of complex arithmetic? The evidence suggests that for certain domains, the phase isn't just noise—it's the signal.

In medical imaging, Elizabeth K. Cole demonstrated in Complex-Valued Convolutional Neural Networks for MRI Reconstruction that CVNNs outperform real-valued models because they respect the algebraic structure of the raw data. Similarly, in Embracing the Imaginary, Erika Bondareva showed that heart murmur detection becomes significantly more robust when the model can "hear" the phase relationships in the acoustic signal.

Even in computer vision, Lai Jiang found in Image Saliency Prediction in Transformed Domain that the phase spectrum of an image contains the most robust cues for human visual attention. When standard non-linearities destroy this phase, the model loses its ability to "see" the structural skeletons of the scene.

The Future: Gating and Hypercomplex Spaces

The field is now moving toward more sophisticated control of these complex signals. Moritz Wolter and Angela Yao developed Complex Gated Recurrent Neural Networks, which use complex-valued gates to regulate information flow. Instead of just turning a neuron "on" or "off," these gates can shift the phase or scale the magnitude, providing a much richer vocabulary for sequence modeling.

Beyond complex numbers, researchers like H. Henry Cao are exploring hypercomplex spaces (quaternions and octonions) in models like Numerion, applying the same principle: use the norm to introduce non-linearity, and use the high-dimensional "phase" to preserve the multidimensional relationships within the data.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_phase_preserving_non_linearities_in_comp_be7f3f95,
  title = {Phase Preserving Non-linearities in Complex Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/phase-preserving-non-linearities-in-complex-neural-networks/rdr_be7f3f952d15478b88c82f94009f9581},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_phase_preserving_non_linearities_in_comp_be7f3f95_v1,
  title = {Phase Preserving Non-linearities in Complex Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/phase-preserving-non-linearities-in-complex-neural-networks/rdr_be7f3f952d15478b88c82f94009f9581/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}