Introduction
Modern language modeling is currently defined by a fundamental trade-off between efficiency and effectiveness. While Transformers, utilizing Quadratic Causal Attention, excel at capturing long-range dependencies, they suffer from $O(L^2)$ complexity, which limits their scalability for long sequences. Conversely, State Space Models (SSMs) like Mamba2 offer linear scaling ($O(L)$) but often struggle with information loss due to their fixed-size hidden states.
This paper introduces Cheems, a "Wonderful Matrix" architecture designed to unify these paradigms. Cheems integrates the linear-scaling efficiency of SSMs with the high expressivity of Transformers through four key innovations: Unified RoPE for SSD, Inner Function Attention (IFA), Dynamic Attention Masks, and Cross-Domain Mixture of Experts (CDMoE).
The Cheems Architecture
The core philosophy of Cheems is to create a hybrid model where different components share a consistent internal representation, particularly regarding positional information and expert knowledge.
1. Unified RoPE for SSD
Rotary Position Embeddings (RoPE) have become the standard for Transformers but lacked theoretical validation for State Space Duality (SSD) algorithms. The authors provide a mathematical proof demonstrating that RoPE is compatible with SSD. This allows Cheems to maintain a single, consistent positional encoding across both its attention and SSM layers, eliminating the need for redundant causal convolutions.
2. Inner Function Attention (IFA)
Standard attention mechanisms use a simple linear projection for the "Value" ($V$) vector. Cheems introduces Inner Function Attention (IFA), which replaces this projection with a heuristic function utilizing a top-k retrieval mechanism. This allows the model to access a larger pool of shared value states, significantly increasing the expressive capacity of the hidden state without the computational explosion associated with standard KV caches.
3. Dynamic Attention Mask
To mitigate "sequence noise" in long contexts, Cheems employs a learnable, dynamic mask. Unlike static causal masks, this plug-and-play component learns to selectively filter information during attention calculations, functioning similarly to the gating mechanisms found in SSMs to focus on relevant context.
4. Cross-Domain Mixture of Experts (CDMoE)
Traditional Mixture of Experts (MoE) models often suffer from parameter redundancy. CDMoE addresses this by utilizing shared "cross-domain" parameters for general knowledge, while reserving "private" experts for specialized data. This architecture allows for finer granularity and more experts without sacrificing retrieval speed or increasing the overall parameter footprint unnecessarily.
Key Results and Performance
Cheems was evaluated against industry standards including Llama, Mamba2, and Jamba at the 1.3B parameter scale. Pre-trained on the Smollm-Corpus and Chinese Cosmopedia, the model demonstrated superior performance across several dimensions:
- Benchmark Superiority: Cheems achieved an average score of 52.44 across major benchmarks (MMLU, TriviaQA, ARC, etc.), outperforming Llama-1.3B (50.36) and Jamba-1.3B (51.07).
- Throughput: On RTX 4090 GPUs, Cheems showed significantly higher throughput than standard Llama and Jamba architectures, nearly matching the speed of pure-SSM models like Mamba2.
- Long-Context Robustness: In Multi-Query Associative Recall (MQAR) tasks, the combination of IFA and Dynamic Masking allowed Cheems to maintain high accuracy at sequence lengths of 2048, where standard SSD and attention models began to degrade.
Conclusion
Cheems represents a significant step toward a unified architecture that does not compromise on either speed or intelligence. By proving the compatibility of RoPE with SSD and rethinking the role of the Value projection through IFA, the authors have created a model that is both more efficient and more effective than its predecessors. Future work involves exploring the "Doge" architecture—a variant that investigates whether a purely optimized Attention/CDMoE stack can achieve similar results without the SSD module.
Summary updated