Complex Language Models as Two-Dimensional Hypercomplex Networks

Complex-valued language models represent each word as a set of complex numbers, where the length of each number carries how strongly a feature is present and the angle (the "phase") carries things like position or polarity. Such models keep reporting small gains in translation and language modeling. They also carry a reputation for being expensive: a complex multiplication costs four real multiplications, and the favored nonlinearities need square roots and arctangents. This note argues that the expense is largely an accounting artifact. A complex linear layer is exactly a hypercomplex weight-sharing layer of order two. At the same number of real floating-point operations (FLOPs) as a real layer, it uses half the free parameters. It can run as one ordinary real matrix multiply on standard GPU hardware. Once complex layers are counted this way, the only real overhead left is the nonlinearity. Approximation theory tells us which nonlinearities are allowed. We propose a cheap, square-root-free activation that preserves phase and satisfies the theory's condition. We then use it to answer a question the literature has not asked cleanly: is the complex algebra itself useful for language, or would any learned two-dimensional weight-sharing rule do just as well?

In short, we build a fully complex, causal Transformer whose layers are implemented as structured real matrix multiplies and whose activation is a gate on $|z|^2$. We compare it against real, learned-algebra, and quaternion variants at matched FLOPs on WikiText-103 and text8. The goal is to separate three possible sources of benefit: parameter tying, the specific rotation structure of complex numbers, and the choice of nonlinearity.

Core Claim

A complex linear map $W = A + iB$ acting on $z = x + iy$ equals the real block matrix $\begin{pmatrix} A & -B \\ B & A \end{pmatrix}$ acting on $[x; y]$. This is the parameterized hypercomplex multiplication $W = \sum_{i=1}^{2} A_i \otimes F_i$ used in PHNNs and Compacter, with the two $2\times 2$ "algebra" matrices $A_i$ frozen to the identity and to a 90-degree rotation. The "4x arithmetic tax" documented for complex networks by Zhang et al. and Gu et al. compares a complex layer against a real layer of the same logical width. Against a real layer of the same real width, the complex layer costs identical FLOPs and stores half the parameters.

The same logic reaches into attention. The attention score $\mathrm{Re}(q^\top \bar{k})$ is a real dot product of concatenated real and imaginary parts. It is therefore compatible with fused real attention kernels and with causal masking. That compatibility avoids the global FFT mixing in PRISM, which does not support autoregressive decoding.

What remains is the activation. Voigtlaender's universal approximation theorem gives the conditions for deep complex networks to approximate any continuous function. The activation must not agree almost everywhere with a polynomial in $z$ and $\bar z$, and it must not be holomorphic or antiholomorphic ("holomorphic" meaning complex-differentiable, which rigidly ties the real and imaginary parts together). Geuchen and Voigtlaender show that smooth non-polyharmonic activations achieve the optimal approximation rate. A function is polyharmonic if repeatedly applying the Laplacian eventually gives zero. The optimal rate is error $O(m^{-k/(2n)})$ for $m$ neurons, smoothness $k$, and $2n$ real input dimensions. modReLU is known to satisfy these conditions, but it pays for a square root, a division, and implicitly an angle. We propose

$$\sigma(z) = z \cdot \phi\big(|z|^2 + b\big),$$

where $|z|^2 = x^2 + y^2$ costs two multiplies and an add, $b$ is a learned per-channel bias, and $\phi$ is either a hard sigmoid (clamp) or a smooth sigmoid. The gate multiplies $z$ by a real number, so the phase of $z$ passes through unchanged. It also commutes with any global phase rotation $e^{i\alpha}$, as modReLU does and the cardioid does not.

The hard-sigmoid version has kinks, so it is not real-analytic. Polyharmonic functions are always real-analytic, so it cannot be polyharmonic, and its kinks also rule out being holomorphic. It therefore satisfies Voigtlaender's deep universality condition directly. For the smooth version, which is the case Geuchen and Voigtlaender's rate theorem covers, we will prove non-polyharmonicity. The argument reduces $\Delta^k[z\,h(r^2)] = 0$ to an Euler-type ordinary differential equation in $r^2$. The solutions of that equation are powers and logarithms, and a sigmoid is neither.

The empirical claim is concrete. A complex causal Transformer using this gate should reach the perplexity of a real Transformer with the same FLOPs while using about half its non-embedding parameters. It should run within roughly 1.2x of that real model's wall-clock time, rather than the 3x slowdown Quaternion Recurrent Neural Networks report for quaternion LSTMs without custom kernels.

Research Program

Unified implementation. We implement every linear map in the model as one parameterized family, $W = \sum_{i=1}^{n} A_i \otimes F_i$, executed as a single dense real GEMM after the block matrix is assembled. The family covers five points:

  • $n=1$: the real model.
  • $n=2$ with $A_i$ frozen to the complex rules.
  • $n=2$ with $A_i$ learned, as in PHM layers.
  • $n=4$ frozen to the Hamilton product.
  • $n=4$ learned.

Assembling the block matrix is cheap relative to the multiply, so all variants at equal real width have equal FLOPs and differ only in parameter count and tying structure. This addresses the kernel-mismatch problem described by Arjovsky et al. and the autograd tensor-splitting problem described by Oh et al.. It does so by never exposing complex tensors to the autograd engine at all.

Embeddings. The word embedding follows Encoding word order in complex embeddings: word $j$ at position $pos$ in dimension $d$ is $r_{jd}\,e^{i(\omega_d\, pos + \theta_{jd})}$. The amplitude $r$ encodes the word's lexical content, the frequency $\omega$ controls sensitivity to position, and the offset $\theta$ is a starting phase. Shifting position then becomes a pure rotation of phase. We share frequencies across words, which gives a table exactly the size of a real table of width $2D$. We ablate the per-word-frequency variant that paper uses, which costs $3 \times D \times |W|$. That paper found frequency sharing costs accuracy. We test whether this still holds once the downstream layers are also complex, rather than projected back to real numbers.

Theory. Beyond the smooth-gate lemma, we check which commonly used "complex" activations actually fail Voigtlaender's conditions. Complex tanh and low-degree polynomials in $z$ are holomorphic on their domain. We include them deliberately as activations the theory predicts should fail. Split activations, which apply a real function separately to the real and imaginary parts, are non-polyharmonic but break phase equivariance. They separate "universal" from "phase-preserving." The activation ablation is therefore a small test of whether the theory's distinctions predict training behavior in deep language models, which the file shows no one has tested; the theory papers contain no experiments.

Phase analysis. We repeat PRISM's intervention in a causal model: we scramble either the phase or the magnitude of hidden states and measure the resulting perplexity change. PRISM reports a 72–205x degradation from phase scrambling and 1.1x from magnitude scrambling. Two outcomes are possible. If phase is equally dominant in our model, then phase-destroying split activations should lose. If it is not, PRISM's finding may reflect its FFT-convolution architecture rather than complex language models in general.

Baselines

The naive reference is a standard real pre-norm Transformer with rotary position encoding (RoPE, which rotates query and key vectors by position-dependent angles) and GELU activations. It is shrunk to match the complex model's parameter count. This is the "do nothing" option: fewer parameters, obtained by narrowing a real model. Beating it by a clear margin, more than the seed-to-seed spread across three seeds, is the minimum bar.

The second real baseline has the same real width as the complex model, and therefore about twice its non-embedding parameters and equal FLOPs. This is the ceiling we aim to match. We count success as reaching its perplexity within 1% using half the parameters.

The strongest competitor on the algebra question is the learned $n=2$ parameterized hypercomplex model. It uses the Kronecker weight sharing of PHNNs and Compacter, applied to every layer rather than to adapters, and has exactly the complex model's parameters and FLOPs. Compacter reports a GLUE average of 86.47 against 86.50 for full fine-tuning on T5-base. That shows learned-algebra tying costs little accuracy, although in a fine-tuning regime rather than pretraining. The complex model "beats" this baseline only if the frozen complex rule gives lower perplexity at the same activation. It must also do so with a phase-preserving activation, which a learned algebra has no principled way to support.

The strongest existing complex language model in the file is PRISM's Wave-Particle Transformer. It reports WikiText-103 perplexity 4.94 against 5.28 for its Transformer baseline (Language as a Wave Phenomenon). These numbers come from a different tokenization and setup than ours, so we will not compare against them directly. We report our own reproduced baselines. If PRISM code is available, we will run it on our data and token budget.

On text8, the file reports 1.26 bits per character for the complex-order model against 1.29 for Transformer-XL (Encoding word order in complex embeddings). We treat these as the reference numbers for that benchmark.

The key ablation isolates the activation. The same complex model is trained with each of five activations: the $|z|^2$ gate, modReLU, the cardioid, split GELU, and complex tanh. For every comparison, all models get the same token budget, the same learning-rate sweep of five values, and the same number of GPU-hours. We report the best configuration per model.

Why not just…?

Why not just use a narrower real Transformer, or a factorized embedding table? This is the simplest way to cut parameters, and it is the naive baseline above. The evidence against it comes from other modalities. On TIMIT, a quaternion LSTM reaches 15.1% phoneme error rate with 14.4M parameters, versus 15.3% for a real LSTM with 46.2M (Quaternion Recurrent Neural Networks). We have no language-modeling evidence yet, which is exactly why this baseline is run first. If the narrow real model matches the complex model, the complex parameterization buys nothing, and the proposal fails at step one.

Why not just learn the two-dimensional algebra with PHM and skip complex numbers? This is the real threat. Learned tying is strictly more flexible, and it may absorb whatever the frozen complex rule offers. The learned-$n=2$ baseline settles the question directly. We will also inspect whether the learned $A_i$ drift toward a rotation. If learned PHM matches or beats the complex model, we will report that the benefit comes from tying rather than from complex structure. The activation result still stands in that case, but only for phase-preserving models.

Why not just keep modReLU and accept its cost, or use split activations? modReLU is proven non-polyharmonic (Optimal approximation using complex-valued neural networks), so it is a fine baseline. Our gate must match its perplexity while measurably reducing activation-layer latency, or it is not worth adopting. Split activations are cheaper still and also universal. If they match the phase-preserving gates, then either phase carries less information in causal language models than PRISM's interventions suggest, or equivariance is unnecessary. The phase-scrambling experiment tells these apart.

Why not just concatenate real and imaginary parts into a wider real model? At matched FLOPs, that is the equal-width real baseline. The only direct evidence is on non-text signals: Complex Transformer reports 74.22% versus 71.30% average precision on MusicNet, and 59.94% versus 57.57% accuracy on radio IQ signals, for complex versus concatenated models. Our experiment is the text counterpart of that comparison.

Proposed Experiments

The main grid trains models at roughly 50M and 150M real-equivalent width on WikiText-103, measuring validation and test perplexity, and on text8, measuring bits per character. The grid crosses the five algebra variants with the five activations. We report parameters, FLOPs, tokens per second, peak memory, and time per step for every run.

Three results decide the proposal:

  1. Parameters. The complex model with the $|z|^2$ gate should land within 1% of the equal-width real model's perplexity using half its parameters. It must also beat the parameter-matched narrow real model.
  2. Theory versus training. Holomorphic activations should train measurably worse than the non-polyharmonic ones. That would confirm Voigtlaender's distinction predicts behavior in deep language models.
  3. Speed. The GEMM-based implementation should keep wall-clock overhead below 1.2x relative to the real model at equal width. The main remaining cost should come from the gate, not the matrix multiplies.

As a secondary probe of the claim that phase encodes word sense, we will freeze the pretrained model and train a linear probe on phase angles for word sense disambiguation on SemCor. QPFE-ERNIE reports an average F1 gain of 5.2 points over BERT on this benchmark. We will also run the PRISM-style phase- and magnitude-scrambling intervention on the complex model.