TL;DR
This paper introduces ComplexFormer, a novel Transformer architecture that enhances attention mechanisms by treating queries and keys as complex numbers. By allowing each attention head to independently learn how to integrate semantic and positional information through complex rotations, ComplexFormer achieves state-of-the-art performance in language, code, and math tasks with improved parameter efficiency.
What Problem Does This Paper Address?
Transformers, the backbone of modern AI, rely on self-attention to understand relationships between tokens in a sequence. However, effectively incorporating positional information (where a token is in the sequence) alongside semantic information (what the token means) has been a persistent challenge. Existing methods often treat these two aspects separately or apply a uniform positional encoding strategy across all attention heads. This uniformity can limit the ability of different heads to specialize in capturing unique patterns, potentially hindering the model's overall representational capacity and performance, especially in tasks requiring understanding long-range dependencies.
Methods
The core innovation of ComplexFormer is its Complex Multi-Head Attention (CMHA). Instead of standard real-valued operations, CMHA treats query and key vectors as complex numbers. This is achieved through two key mechanisms:
- Per-Head Euler Transformation: Real-valued query and key projections are converted into polar-form complex vectors. This transformation allows for operations in a complex subspace, where interactions can be represented as rotations and scaling.
- Per-Head Adaptive Differential Rotation: This is the most significant departure. Each attention head learns its own unique way to combine semantic information (represented as an angle difference, AS) with relative positional information (represented as a positional angle, ΔP). This is done via a learned, head-specific function
Adapti(ASmn,i) + ∆Pmn,i. This allows each head to develop its own strategy for integrating these two crucial pieces of information, moving beyond the uniform approach of methods like RoPE.
Essentially, CMHA allows each attention head to become a specialized "complex number manipulator," learning how to best rotate and scale information based on both its meaning and its position, tailored to its specific role within the attention mechanism.
Datasets
The paper evaluates ComplexFormer on a variety of standard benchmarks:
- Text Generation: WikiText-103
- Code Generation: The Stack (filtered subset of Python/Java), HumanEval, MBPP
- Mathematical Reasoning: GSM8K, MATH
Key Insights and Ideas
- Head Specialization is Key: The most impactful finding is that allowing each attention head to independently learn how to integrate semantic and positional information leads to significant performance gains. This contrasts with prior methods that applied uniform strategies.
- Complex Numbers as a Natural Fit: The paper demonstrates that complex numbers, with their inherent representation of magnitude and phase (angle), provide a powerful and unified framework for modeling both semantic relationships and positional dependencies.
- Adaptive Rotation Unlocks Potential: The "adaptive" part of the differential rotation is crucial. By learning specific rotation parameters for each head, ComplexFormer can better capture nuanced interactions that a fixed or shared approach would miss.
- RoPE as a Special Case: The paper positions RoPE as a simplified, non-adaptive component of their proposed CMHA, highlighting the generality and advancement of their approach.
Key Results
ComplexFormer consistently outperforms strong baselines, including RoPE-based Transformers, across various tasks:
- Text Generation: Achieved the lowest generation perplexity (34.2) on WikiText-103, indicating improved fluency and coherence.
- Code Generation: Showed superior performance on both HumanEval and MBPP benchmarks, with higher Pass@k scores, suggesting better understanding of code structure.
- Mathematical Reasoning: Achieved the highest accuracy on GSM8K (21.3%) and MATH (5.2%) test sets.
- Parameter Efficiency: ComplexFormer achieved these state-of-the-art results with comparable or fewer parameters than larger models like GPT-2 Base, demonstrating its efficiency.
Limitations and Future Work
The authors acknowledge several limitations:
- Computational Overhead: While asymptotic complexity remains the same as standard MHA, the explicit complex number operations introduce a constant factor overhead. This could become more pronounced in extremely large models.
- Parameter Count for Adaptation: The adaptive parameters, while marginal for current model sizes, could become more noticeable in very deep or wide models. Parameter sharing schemes are suggested as a potential mitigation.
- Interpretability: Understanding precisely what each head learns through its adaptive parameters (δ, b) is challenging and requires further analysis.
- Long-Context Generalization: While improvements suggest better coherence, specific evaluations on extremely long sequences (beyond 4096 tokens) are needed to fully characterize its capabilities and limits.
Future work includes scaling ComplexFormer, applying it to other modalities (vision, speech), analyzing head specialization, and exploring alternative complex transformations and adaptive functions.
Summary updated