Phase Where It Counts: A Compute-Matched Test of What Complex Numbers Buy Language Models

Several recent language models replace ordinary real-valued vectors with complex numbers. Each coordinate then carries a size and an angle (its "phase"), and the models report better perplexity or reasoning scores than real-valued Transformers. Complex arithmetic is not free, though. One complex multiply costs four real multiplies, and complex layers run on GPU software built for real matrices. In speech enhancement, a careful audit found that a complex network needed about three times the compute of a real one to reach the same quality. No comparable audit exists for language models. So we do not know whether their gains come from something only complex numbers provide, or from a network that is effectively twice as wide and doing three to four times the work.

This proposal tests one specific hypothesis. The useful part of complex structure in language models is cheap: it is content-dependent rotation of token representations, which costs a number of operations linear in the width. The expensive part, dense complex weight matrices, is mostly redundant with simply widening a real network. We will build a "phase-sparse" Transformer that keeps every dense matrix real and uses complex numbers only for a small learned rotation applied to queries, keys, and the residual stream. We will compare it against full complex language models and width-matched real models under equal compute, equal wall-clock time, and equal tuning budgets, at several model sizes. If the hypothesis holds, the field gets most of the complex benefit at near-zero overhead. If it fails, we learn which dense complex operations actually earn their cost.

Core Claim

The claim is that, at matched compute, a real Transformer with a content-conditioned phase rotation matches or beats both a fully complex language model and a real model given the same compute budget. The claim also has a mechanistic side: the rotation angles themselves should carry context-dependent meaning, detectable as larger rotations on idiomatic uses of an expression than on literal uses.

The motivation comes from three findings in the file. Language as a Wave Phenomenon builds a complex spectral model, PRISM, in which scrambling the phase degrades performance 72–205x while scrambling magnitudes costs only 1.1x. It also finds that ambiguous words rotate more in deeper layers. The information seems to live in the angle. ComplexFormer shows that RoPE, the rotary position encoding used in LLaMA-style models, is a special case of its per-head adaptive rotation, in which each head learns how token meaning, not just token distance, sets the rotation angle.

Both papers pay for their rotations with fully complex projections, and neither reports compute-matched real baselines. Rethinking Complex-Valued Deep Neural Networks for Monaural Speech Enhancement supplies the constraint. It reports 14.38 G multiply-accumulate operations (MACs) for complex DCCRN against 4.84 G MACs for a real counterpart with essentially identical quality, and complex models falling further behind as parameter budgets shrink. The Computational Cost of Complex Valued Neural Networks explains the cost: the four-multiply arithmetic tax plus the overhead of software that tracks real and imaginary parts separately. Our design takes phase as the thing worth keeping and dense complex algebra as the thing to question.

The Phase-Sparse Transformer

The model is a standard causal decoder with real weights throughout. Each head's query $q$ and key $k$ (real, width $d$) are read as $d/2$ complex numbers by pairing coordinates. RoPE does this already. A small real linear map $P$ of rank $r \ll d$, applied to the token's current hidden state $x$, produces one angle per complex coordinate, $\phi(x) = P x$. The attention score between positions $m$ and $n$ is

$$ s_{mn} = \mathrm{Re}\sum_{j=1}^{d/2} q_{m,j}\,\overline{k_{n,j}}\; e^{\,i\left(\phi_j(x_m) - \phi_j(x_n) + \omega_j (m-n)\right)}, $$

where $\omega_j$ are the fixed RoPE frequencies and the bar denotes complex conjugation. The same content-conditioned rotation, without the position term, is applied once to the residual stream before each feed-forward block. That gives phase a place to accumulate context across layers, which is the effect PRISM measures.

The extra cost is about $d \cdot r$ MACs per token plus $d$ sine and cosine evaluations. A layer's dense projections cost roughly $d^2$. The rotation is a per-coordinate 2×2 operation, so it fuses into existing RoPE kernels and FlashAttention-style attention without new complex kernels. This is the key difference from ComplexFormer, which makes the query and key projections themselves complex. It is also why the design sidesteps the missing fused-kernel problem the file identifies, rather than trying to solve it.

Research Program

The work has three parts. The first is a size ladder: four model sizes on WikiText-103 causal language modeling, from small models (where the speech paper predicts complex models are hurt) up to the largest budget we can afford. This directly probes the unresolved tension between the speech findings and Unitary Evolution Recurrent Neural Networks. That paper reports a 9k-parameter complex recurrent network beating a 60k-parameter LSTM on permuted-pixel MNIST (91.4% vs 88%). Our reading is that the unitary network's advantage came from a structured rotation constraint, not dense complex capacity, which is exactly the component the phase-sparse model keeps.

The second part is accounting. For every model we report parameters, analytic MACs, measured training throughput, peak memory, and inference latency on the same GPU. Complex baselines are run both with native complex tensors and with explicit real expansion, so the arithmetic cost is not unfairly inflated by a poor implementation.

The third part is mechanism. We scramble learned phases at inference time, following the PRISM intervention. We also probe whether rotation magnitude separates idiomatic from literal uses in MAGPIE, which contains 56,622 annotated instances of 1,756 potentially idiomatic expressions. The motivating literature, from Quantum-Inspired Complex Word Embedding to CNM, claims that phase captures this kind of non-additive meaning, but no one has tested it on an idiom corpus. CNM also warns that unsupervised phases often lack clear linguistic meaning, so the probe could come out negative. That is why it is a check rather than an assumption.

A secondary evaluation on SciTail entailment tests the asymmetric-matching argument from Hermitian Co-Attention Networks. The phase-sparse score keeps an asymmetry, because each side's rotation depends on its own token. Measuring this tells us whether that asymmetry is enough.

Baselines

The naive reference point is a real decoder with RoPE at the same depth and width. It is the "do nothing" model, and every other number is read against it.

The second and most dangerous baseline is a width-matched real model. We widen the RoPE decoder until its measured training compute equals that of the fully complex baseline, and separately until its parameter count matches. This is the concatenated-real argument made concrete. The speech audit suggests it will often tie complex models, and on MusicNet Complex Transformer beat its concatenated real counterpart only modestly (74.22% vs 71.30% APS).

The strongest existing method is a ComplexFormer-style model with complex query and key projections and per-head adaptive rotation. ComplexFormer reports a WikiText-103 generation perplexity of 34.2. We reimplement it at each rung of the size ladder rather than trusting cross-paper numbers, because tokenization and model size differ.

PRISM's reported 4.94 against 5.28 perplexity comes from a non-causal model. It is not comparable to next-token perplexity, so we cite it only as evidence for the phase mechanism, not as a target number. For SciTail, the file reports only HCRN's roughly 10% gain over ESIM and no absolute accuracy, so there is no reported result found on Lacuna for an absolute SciTail score. We will rerun ESIM-style and HCRN-style heads on our backbones.

All models share data, tokenizer, schedule family, and an identical hyperparameter search budget, with three seeds each. Two ablations isolate the key ingredient. In the first, the rotation generator $P$ is frozen at zero, which reduces the model exactly to RoPE. In the second, the angles are computed from a random fixed projection instead of a learned one. Beating the baselines means lower validation perplexity than both the width-matched real model and the ComplexFormer-style model at equal measured compute, with the gap exceeding seed variance at at least three of the four sizes, and with overhead above the RoPE baseline under 5% in wall-clock time.

Why not just…?

Why not just double the real width and forget complex numbers? This is a real threat, and the width-matched real baseline tests it directly. Circularity arguments from radio-frequency work, such as Enhancing RF Sensing with Deep Learning, concern physical wave signals, not tokens, so they do not transfer automatically to language. If the widened real model matches the phase-sparse model at equal compute, we will report that complex structure brings nothing to language modeling at these scales. That negative result would still be useful, given how many complex language models skip this comparison.

Why not just use RoPE, which already rotates? RoPE's angle depends only on distance. The frozen-generator ablation is exactly RoPE inside our model, so the gap between it and the learned-generator model measures the value of content-dependent phase alone. ComplexFormer's gains suggest that gap is non-zero, but that evidence confounds rotation with dense complex projections. Our design separates the two.

Why not just write a fused complex attention kernel and keep full complex models? A kernel removes software overhead but not the arithmetic itself. Complex matrix products still need three or four real multiplies per complex multiply-add, per the computational-cost analysis. We measure complex baselines both with native complex tensors and with explicit real expansion to bound how much a kernel could recover. If a full complex model beats the phase-sparse model even after that best-case reduction, dense complex algebra earns its place, and we will say so.

Why not just rely on real attention's asymmetry from separate query and key matrices? Separate projections already make real attention asymmetric. That undercuts the HCRN argument for full Transformers, though not for HCRN's shared-encoder matching layer. The SciTail comparison between the RoPE baseline, the phase-sparse model, and a Hermitian head settles whether complex asymmetry adds anything once projections are untied.

Decisive Checks

Three outcomes decide the proposal.

The first is compute efficiency. At equal measured compute, the phase-sparse model should reach lower perplexity than the width-matched real model, and within seed noise of or below the ComplexFormer-style model, across the size ladder. The advantage should not vanish at the smallest size. If it does vanish there, the speech paper's small-model pathology extends to language, even in this cheaper form.

The second is phase scrambling. Randomizing the learned angles at inference should hurt the phase-sparse model far more than randomizing an equal amount of the real dimensions. If it does not, the model is not using phase and the gains are an artifact.

The third is the idiom probe. On MAGPIE, the per-token rotation of an expression's words should be larger for idiomatic instances than for literal instances of the same expression, and a linear classifier on rotation angles should beat one given magnitudes alone. A negative result here would not sink the efficiency claim. It would, however, refute the idea that phase carries compositional meaning, which is the linguistic story behind the whole line of work, and we will report it either way.