iFairy: the First 2-bit Complex LLM with All Parameters in {±1, ± i}

Feiyu Wang, Guoan Wang, Yihao Zhang, Shengfan Wang, Weitao Li, Bokai Huang, Shimao Chen, Zihan Jiang, Rui Xu, Tong YangarXiv 2025 · DOI · arXiv

This paper introduces iFairy, a novel framework that represents the first successful attempt to build a 2-bit Large Language Model (LLM) using complex-valued parameters. By shifting from the real number domain to the complex plane, the authors "raise the ceiling" of what low-bit models can achieve, outperforming existing state-of-the-art 2-bit models and even rivaling full-precision benchmarks.

TL;DR

iFairy is the first 2-bit complex-valued LLM where all weights are mapped to the fourth roots of unity ($\{\pm 1, \pm i\}$). By leveraging the richer representational space of complex numbers, it breaks the performance limits of traditional real-valued quantization while remaining entirely multiplication-free during inference.

What Problem Does This Paper Address?

Deploying LLMs in the real world is hindered by two major bottlenecks: spatial (the massive memory footprint of billions of parameters) and temporal (the slow, power-hungry nature of large-scale matrix multiplications).

To solve this, researchers use Quantization-Aware Training (QAT) to compress models into ultra-low bitwidths (like 2-bit). However, all existing QAT methods are limited by a fundamental "accuracy ceiling": a quantized model can, at best, only hope to minimize its error relative to a full-precision real-valued model (like LLaMA).

Current leaders in this space, such as BitNet b1.58, use ternary weights $\{-1, 0, 1\}$. While efficient, this approach doesn't fully utilize the 2-bit capacity (using only 1.58 bits of entropy) and is constrained by the representational limits of the real number line. iFairy asks: What if we change the number system itself to make the "ceiling" higher?

Methods

The authors re-engineer the Transformer architecture to operate entirely in the complex domain ($\mathbb{C}$).

  1. Complex-Valued Backbone: Every component—from embeddings to the feed-forward networks—is adapted for complex numbers. This includes a Dual-channel Projection for embeddings and a Complex-Valued Self-Attention mechanism that uses the real part of the Hermitian inner product to calculate attention scores.
  2. PhaseQuant: This is the core quantization algorithm. It maps full-precision complex weights to the set $\{\pm 1, \pm i\}$ based on their phase in the complex plane.
  3. Multiplication-Free Inference: Because the weights are restricted to $\{\pm 1, \pm i\}$, multiplying an activation by a weight doesn't require a floating-point multiplication. It only requires sign flips (for $\pm 1$) or swapping the real and imaginary parts (for $\pm i$). This transforms matrix multiplication into an addition-intensive process, which is much faster on hardware.
  4. Squared ReLU: To maintain non-linearity in the complex domain while keeping operations computationally inexpensive, the model uses a squared ReLU activation function applied independently to real and imaginary components.

Datasets

The models were trained from scratch on a 100B-token subset of the RedPajama-V1 dataset.

Evaluation was performed on:

  • Language Modeling: WikiText2 and C4 (measuring Perplexity).
  • Downstream Tasks: A suite of zero-shot reasoning benchmarks including ARC-Easy, ARC-Challenge, Hellaswag, Winogrande, and PIQA.

Key Insights and Ideas

  • Breaking the Ceiling: The most significant insight is that a complex-valued architecture is inherently more expressive than a real-valued one of the same scale. By "raising the ceiling" of the full-precision model, the resulting 2-bit quantized model starts from a higher baseline of accuracy.
  • Information-Theoretic Optimality: While ternary models (BitNet) use 3 states, iFairy uses all 4 states available in a 2-bit budget ($\log_2(4) = 2$ bits). This maximizes the entropy and representational capacity of the weights.
  • Geometric Symmetry: The fourth roots of unity are equidistant on the unit circle. This symmetry provides geometric robustness and leads to more stable optimization dynamics compared to real-valued quantization schemes that often struggle with "representational collapse."
  • Phase over Magnitude: In complex networks, the phase (direction) often carries more critical information for learning than the magnitude. PhaseQuant prioritizes preserving this directional information.

Key Results

iFairy was tested at 700M and 1.3B parameter scales, showing dominant performance over the current 2-bit standard, BitNet b1.58.

  • Superior Accuracy: The 1.3B iFairy model achieved an average downstream task accuracy of 46.52%, outperforming BitNet b1.58 (45.39%) and even slightly surpassing the full-precision FP16 LLaMA benchmark (46.21%).
  • Lower Perplexity: On the C4 dataset, the 1.3B iFairy model reached a perplexity of 10.94, significantly better than BitNet's 11.29.
  • Efficiency: Despite the higher accuracy, iFairy maintains the same storage requirements as BitNet and remains multiplication-free, making it highly suitable for edge device deployment.

Limitations and Future Work

  • Hardware Optimization: While the math is multiplication-free, current GPUs and CPUs are heavily optimized for standard real-valued floating-point math. Realizing the full speedup of iFairy will require custom hardware kernels or specialized accelerators.
  • Scaling: The study only goes up to 1.3B parameters. Whether these complex-valued benefits hold or expand at the 70B+ scale remains an open question.
  • Attention Formulation: The authors note that the "optimal" way to handle complex-valued attention is still an active area of research; their current "real-part" approach is effective but perhaps not the final word.
  • Scaling Factors: The model currently uses separate scaling factors for real and imaginary parts, which might slightly distort the original magnitude structure of the complex weights.

Summary updated

BibTeX
@misc{wang2025ifairy,
  author = {Feiyu Wang and Guoan Wang and Yihao Zhang and Shengfan Wang and Weitao Li and Bokai Huang and Shimao Chen and Zihan Jiang and Rui Xu and Tong Yang},
  title = {{iFairy: the First 2-bit Complex LLM with All Parameters in \{±1, ± i\}}},
  year = {2025},
  eprint = {2508.05571},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.05571},
}
Figure