Is Phase Worth Its Price in Language Models?

Complex-valued word embeddings represent each word as a list of complex numbers, so every dimension has a size (amplitude) and an angle (phase). Their appeal for language is easy to state. Amplitude might carry how strongly a word expresses a concept, and phase might carry context, polarity, or which sense of an ambiguous word is meant. When such vectors are combined, phases can reinforce or cancel each other. That gives a natural model of words whose meanings combine non-additively.

The reported results are encouraging but hard to interpret. Complex embeddings have beaten real bag-of-words baselines on sentiment, and one complex-wave language model reports lower WikiText-103 perplexity with fewer parameters. Yet a complex multiply costs four real multiplies. A careful study in speech enhancement found that a real network matched a complex one at one third of the compute. No one has run the equivalent controlled comparison for text.

This note proposes that comparison, built as a dissection rather than a horse race. A complex layer is exactly a real layer whose weights are forced into 2×2 rotation-and-scale blocks. That lets us ask a sharp question: does the specific algebra of rotation help language models, or does any structured, multiplicative, parameter-sharing constraint do just as well at equal compute? We also locate where in the network complex structure pays for its cost. The working hypothesis is that it pays off cheaply in embeddings and position mixing, and does not pay off in the dense layers.

Core Claim

Write a complex weight $w = a + bi$ acting on a pair of real coordinates. Two structured 2×2 blocks with the same parameter count are

$$ W_{\mathbb{C}} = \begin{bmatrix} a & -b \\ b & a \end{bmatrix}, \qquad W_{\mathrm{hyp}} = \begin{bmatrix} a & b \\ b & a \end{bmatrix}. $$

Here $W_{\mathbb{C}}$ is complex multiplication, a rotation plus scaling. $W_{\mathrm{hyp}}$ is "split-complex" multiplication, a hyperbolic boost with no rotations and no periodic phase. Both use two parameters per block and the same number of multiplies.

Our claim has three parts, each testable. First, at matched floating-point operations (FLOPs), a language model that is complex throughout will not beat a well-tuned real Transformer. This would extend the finding of Rethinking Complex-Valued Deep Neural Networks for Monaural Speech Enhancement from speech to text. That paper showed a real DCCRN matching complex DCCRN at 4.84 vs 14.38 G multiply-accumulates, with complex networks falling further behind as models shrink. Second, whatever real benefit exists is concentrated in elementwise phase operations at the embedding and position-mixing stage, which cost $O(d)$ per token. Third, rotation is special: replacing $W_{\mathbb{C}}$ by $W_{\mathrm{hyp}}$, or by a real multiplicative gate, will lose that benefit.

If the third part fails, the "phase" story in this literature reduces to "structured multiplicative interactions help." That would be worth knowing too.

Prior Work and Where This Sits

Three papers motivate the question by reporting gains from complex text representations, all without a compute-matched control.

Quantum-Inspired Complex Word Embedding maps concept strength to amplitude and polarity to phase. It reports 79.8% on MR and 83.3% on SST against weak real baselines (fastText bag-of-words at 76.5%, a real embedding at 80.0%). Pretrained Quantum-Inspired Deep Neural Network for Natural Language Processing concatenates complex features onto ERNIE and reports word-sense-disambiguation F1 gains of 5.2 points over BERT. It never quantifies the added cost, and its complex part never drives the backbone. Language as a Wave Phenomenon builds PRISM, a complex phasor model that mixes tokens with Fourier transforms. It reports 4.94 perplexity against 5.28 for a standard Transformer on WikiText-103, with 18% fewer parameters. Scrambling phase degrades it 72–205x, while scrambling magnitude costs only 1.1x.

That last result is the strongest evidence that phase carries information. However, it shows that PRISM uses phase, not that phase beats an equal-compute real alternative. PRISM also relies on a parallel real attention stream and on bidirectional Fourier mixing, which cannot do causal generation. We do not try to fix PRISM's causality. Instead, we test its central claim inside causal Transformers, where it can be compared fairly.

Two papers supply the method. EulerFormer shows that polar item embeddings with rotation-based attention improve SASRec by 9.93% NDCG@10 on MovieLens-1M at linear cost per token. This is our template for a cheap "phase only at the input and attention" variant. Toward Complex-Valued Neural Networks for Waveform Generation reformulates complex layers as one real block-matrix multiply, cutting autograd nodes by 55–67% and training time by 25%. We use this trick so the complex models are not penalized by bad engineering.

AoE uses complex division only inside a similarity loss. It is complementary, and we leave it out of scope. Unitary Evolution Recurrent Neural Networks supplies modReLU, an activation that rescales magnitude and leaves phase untouched, which we use in the fully complex variant.

Research Program

We train causal decoder-only language models on WikiText-103 at three sizes, roughly 10M, 50M and 150M non-embedding parameters. Small sizes are included specifically to test the capacity inversion that Wu et al. found in speech. We compare seven model families, defined in the Baselines section below. Every model is tuned with the same per-family hyperparameter search budget.

Each family is placed on two frontiers: perplexity against training FLOPs, and perplexity against measured wall-clock time on the same GPU. A parameter-matched comparison is reported only as a third, secondary view. This is the axis on which complex models usually look best, so it should not be the headline.

Frozen representations from the 50M models are then evaluated in three settings. The first is linear-probe sentiment classification on MR and SST, where CE-Mix numbers exist. The second is word sense disambiguation on SemCor/SensEval, the setting where the "phase encodes sense" story predicts the largest gap. The third is a short fine-tuning pass on the same sentiment sets.

Finally, we run interventions on every model. For complex models, we scramble phase and, separately, scramble magnitude, following PRISM's protocol. We apply the matched perturbation to the split-complex model, scrambling the sign or angle of its hyperbolic coordinate. This tests whether phase-scrambling damage reflects meaning stored in rotation, or simply the destruction of any learned 2D coordinate.

Baselines

The do-nothing reference is a standard real causal Transformer with rotary position embeddings, which already rotates real coordinate pairs at the attention step. It is the strongest everyday competitor because it already contains a limited form of phase at negligible cost. We also train a width-scaled version matched to the FLOPs of the fully complex model. Both use the same data, token budget and tuning budget as everything else.

The strongest existing complex method on text is PRISM. On WikiText-103 it reports 4.94 perplexity against 5.28 for its Transformer baseline (Language as a Wave Phenomenon). We reimplement its wave branch in its native masked setting, purely as a reference point, and report it at matched FLOPs against a real masked model. Our causal complex variant cannot be compared to it directly. On sentiment, the reference is CE-Mix at 79.8% (MR) and 83.3% (SST) (Quantum-Inspired Complex Word Embedding). We found no reported result on Lacuna for complex-valued causal language modeling perplexity on WikiText-103.

The fully complex model uses complex embeddings, complex projections in attention and feed-forward layers, and modReLU. It is computed with the block-matrix kernel. The phase-at-input model uses complex polar embeddings and EulerFormer-style rotation in attention, but a real body.

Two ablations isolate the key ingredient. The first is the split-complex model, which uses $W_{\mathrm{hyp}}$ everywhere $W_{\mathbb{C}}$ appears and keeps parameters and FLOPs identical. The second is a real gated model with multiplicative gated linear units, sized to equal FLOPs. This tests whether "interference" is just multiplicative gating.

Beating a baseline means a lower perplexity at equal FLOPs and equal wall-clock time, with the gap exceeding the spread across three seeds. A win only at equal parameters counts as a compression result, not a representational one.

Why not just…?

Why not just make the real Transformer wider? That is exactly the FLOP-matched real baseline, and it may well win. Wu et al. found that outcome in speech. If it wins at every scale, the answer to this note's question is "no," and the paper reports that cleanly. Nothing in the file shows complex text models beating a compute-matched real model, so we cannot rule this out in advance.

Why not just use rotary embeddings, which are already complex multiplication? This is a real threat to the second claim. If phase helps only at the position step, rotary embeddings already capture it. The phase-at-input variant differs in one way: token identity, not just position, receives a learned phase, as in EulerFormer. If it fails to beat the plain rotary Transformer, the conclusion is that learned token phase adds nothing beyond positional rotation.

Why not just use the block-matrix trick and call any 2×2 coupled layer "complex"? The block-matrix form computes complex arithmetic faster; it does not change what the model computes. The split-complex control has the same block shape, parameters and cost but different algebra. So it directly answers whether rotation matters or whether the block structure alone matters.

Why not just use a real gated network? Phase interference and real gating both multiply signals. If the gated linear unit model matches the complex models on perplexity and on word-sense probes, the quantum-inspired framing adds interpretation but no capability.

Decisive Checks

Three outcomes would settle the question.

First, if the fully complex model beats the real Transformer at equal parameters but loses at equal FLOPs, complex arithmetic in text is a parameter-sharing trick. We would also measure whether this loss widens at the 10M scale, as it did in speech.

Second, if the phase-at-input model recovers most of whatever the fully complex model gains, at close to real cost, phase belongs in the embedding and not in the dense layers. That would give a concrete design rule for spending the "complex tax."

Third, if the complex models beat the split-complex and gated controls on SemCor word-sense probing, and phase scrambling hurts them far more than the matched hyperbolic scrambling hurts the split-complex model, then rotation carries sense information that other multiplicative structures do not. That would be the first evidence for this claim that is not confounded by compute or by perturbation size. If instead the split-complex model matches the complex one, the phase interpretations in prior work should be read as stories about structured coupling, not about waves.