Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in {± 1, ± i}

Tong Yang, Feiyu Wang, Xinyu Tan, Bokai Huang, Yihao Zhang, Guoan Wang, Peizhuang Cong, Tong Yang 0003arXiv 2025 · DOI · arXiv

Abstract

As Large Language Models (LLMs) scale, the computational and memory costs of deployment have become prohibitive. While low-bit quantization offers a solution, performance typically collapses at the 1–2 bit level. Fairy2i introduces a novel framework that losslessly converts standard real-valued LLMs into a "widely-linear" complex-valued representation. By leveraging the unique geometric properties of the complex domain and a recursive residual quantization strategy, Fairy2i enables extremely low-bit weights while maintaining performance near full-precision baselines. This approach allows for the reuse of powerful pre-trained checkpoints, bypassing the need for expensive training from scratch.

Introduction

The rapid advancement of LLMs has led to models with tens or hundreds of billions of parameters, creating a significant bottleneck for inference on consumer hardware. Quantization—the process of mapping high-precision weights to a limited set of discrete values—is essential for efficiency. However, current methods struggle at the extreme low-bit regime (1–2 bits). Post-Training Quantization (PTQ) often suffers from severe accuracy degradation at these levels, while Quantization-Aware Training (QAT) typically requires training from scratch, ignoring the vast ecosystem of existing real-valued pre-trained models.

Fairy2i addresses this gap by bridging the divide between real-valued architectures and complex-valued neural networks. Complex numbers provide additional degrees of freedom (magnitude and phase) that can be exploited for more efficient representation.

The Widely-Linear Transformation

The core innovation of Fairy2i is the mathematical "bridge" that allows a seamless transition from the real to the complex domain. Standard complex-linear maps ($y = Ux$) are insufficient to represent all real-linear operations. To achieve a lossless conversion, the authors utilize a "widely-linear" complex map:

$$y = Ux + W\bar{x}$$

where $\bar{x}$ is the complex conjugate of the input. This formulation ensures that any pre-trained real-valued linear layer can be exactly re-parameterized into the complex domain without any initial loss in accuracy.

The transformation process from a real linear layer to a widely-linear complex layer, enabling the reuse of pre-trained weights.

Phase-Aware and Recursive Quantization

Once the model is in the complex domain, Fairy2i applies a specialized quantization scheme designed to maximize representational capacity.

Phase-Aware Complex Quantization

The framework utilizes a compact codebook consisting of the fourth roots of unity: {±1, ±i}. This 2-bit representation is symmetric on the unit circle. Unlike standard real-valued ternary schemes ({+1, 0, -1}), this phase-based approach aligns better with the high-dimensional geometry of attention mechanisms, preserving more information at low bit-widths.

Recursive Residual Quantization

To further bridge the performance gap with full-precision models, Fairy2i employs a recursive strategy. The model quantizes the weight, calculates the resulting residual error, and then quantizes that error using the same method. The final deployed weight is the sum of these low-bit terms. This multi-stage approach allows for incremental refinement of the weight representation.

The recursive residual quantization process, showing how weights are refined across multiple stages (Stage 0 and Stage 1) to minimize error.

Importantly, these terms are data-independent, meaning they can be computed in parallel during inference, avoiding a linear increase in latency while maintaining high accuracy.

Experimental Results

The authors evaluated Fairy2i using the LLaMA-2 7B architecture, continuing pre-training on a 30-billion-token subset of the RedPajama dataset.

Language Modeling Performance

On the C4 validation set, Fairy2i demonstrated superior perplexity (PPL) compared to existing low-bit baselines:

  • Fairy2i-W1 (1-bit effective): Achieved a PPL of 11.03, outperforming the 1-bit Real-Binary baseline (11.75).
  • Fairy2i-W2 (2-bit effective): Achieved a PPL of 7.85, significantly outperforming state-of-the-art 2-bit PTQ methods like AQLM (8.54) and QuIP# (11.01). For comparison, the FP16 baseline is 6.63.

Common Sense Reasoning

Zero-shot accuracy across five standard benchmarks (ARC-Easy, ARC-Challenge, HellaSwag, PIQA, and Winogrande) further validated the approach. Fairy2i-W2 reached an average accuracy of 62.00%, coming remarkably close to the full-precision performance of 64.72%.

Conclusion and Future Work

Fairy2i provides a robust pathway for deploying LLMs at extremely low bit-widths by leveraging the topological advantages of the complex domain. By enabling the conversion of existing real-valued checkpoints, it offers a more practical alternative to training low-bit models from scratch. Future work includes the development of optimized CUDA kernels to fully realize the "multiplication-free" potential of complex operations on GPUs, as well as scaling the framework to larger models like LLaMA-3 70B.

Summary updated

BibTeX
@misc{yang2025fairy,
  author = {Tong Yang and Feiyu Wang and Xinyu Tan and Bokai Huang and Yihao Zhang and Guoan Wang and Peizhuang Cong and Tong Yang},
  title = {{Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in \{± 1, ± i\}}},
  year = {2025},
  eprint = {2512.02901},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.02901},
}
Figure