TransXSSM: A Hybrid Transformer State Space Model with Unified Rotary Position Embedding

Bingheng Wu, Jingze Shi, Yifan Wu, Nan Tang, Yuyu LuoarXiv 2025 · DOI · arXiv

Introduction

TransXSSM introduces a novel hybrid architecture designed to bridge the gap between Transformers and State Space Models (SSMs). While Transformers excel at complex reasoning through global attention, they suffer from quadratic scaling issues. Conversely, SSMs like Mamba offer linear efficiency but often struggle with precise positional reasoning. TransXSSM resolves this "positional encoding inconsistency" by introducing Unified Rotary Position Embedding (Unified RoPE), allowing these two distinct mechanisms to share a common positional language.

The Positional Inconsistency Problem

In sequence modeling, Transformers are permutation-invariant and require explicit positional cues, typically provided via RoPE. SSMs, however, encode position implicitly through their recurrent and convolutional structures. When these layers are combined in hybrid models, a mismatch occurs: the Transformer layers and SSM layers interpret token positions differently. This information gap limits the model's ability to maintain a consistent frame of reference, ultimately degrading performance in complex reasoning tasks.

Proposed Method: Unified RoPE and TransXSSM

The researchers developed two primary innovations to address this challenge:

  1. Unified RoPE: By mathematically extending Rotary Position Embedding to SSMs, the authors ensure "spectral continuity." RoPE represents positions as rotations in high-dimensional space; by applying these same transformations to the state-update signals within the SSM, positional information flows seamlessly across different layer types without being reset.
  2. TransXSSM Block Design: The architecture integrates RoPE directly into both the attention and state-space components, as shown in the comparison below.
Comparison of Transformer, Mamba, and TransXSSM blocks highlighting the integration of RoPE into the State-Space path.

Hybrid Architecture

The TransXSSM model utilizes a specific hybrid stack characterized by a 7:1 ratio. For every seven SSM layers—which handle the bulk of long-sequence processing—there is one Transformer attention layer to provide global context and high-level reasoning. This configuration is supported by modern stability features, including RMSNorm and residual connections.

The TransXSSM macro-architecture illustrating the 7:1 ratio of State-Space to Self-Attention blocks.

Key Insights and Results

The researchers found that the duality between modern SSMs and linear attention allows RoPE to be injected into the SSM recurrence without sacrificing linear-time efficiency. This consistency in the "positional frame of reference" is critical for unlocking the reasoning power of Transformers within an efficient SSM framework.

Performance Metrics

  • Efficiency: At a 4K sequence length, TransXSSM is 42.3% faster to train and 29.5% faster for inference than standard Transformers.
  • Accuracy: The model outperformed Transformer baselines by over 4% on average across language modeling tasks. Notably, on the Winogrande benchmark, TransXSSM-1.3B outperformed Llama3-1.3B by 6.7 points (62.09 vs 55.40).
  • Long Context: In "needle-in-a-haystack" tests, TransXSSM maintained near-perfect retrieval accuracy across long contexts, significantly outperforming pure SSMs that often lose track of precise facts.

Conclusion and Future Work

TransXSSM demonstrates that hybrid models can achieve the "best of both worlds"—the reasoning depth of Transformers and the efficiency of SSMs—provided their positional encodings are unified. While currently tested up to 1.3B parameters, future work will explore whether these scaling advantages persist at the 70B+ parameter frontier and investigate optimal layer ratios for specialized tasks.

Summary updated

Abstract

Transformers exhibit proficiency in capturing long-range dependencies, whereas State Space Models (SSMs) facilitate linear-time sequence modeling. Notwithstanding their synergistic potential, the integration of these architectures presents a significant challenge, primarily attributable to a fundamental incongr inuity their respective positional encoding mechanisms: Transformers rely on explicit Rotary Position Embeddings (RoPE), while SSMs leverage implicit positional representations via convolutions. This divergence often precipitates discontinuities and suboptimal performance.To address this impediment, we propose a unified rotary position embedding (Unified RoPE) methodology, thereby establishing a consistent positional encoding framework for both self-attention and state-space components. Using this Unified RoPE, we introduce TransXSSM, a hybrid architecture that coherently integrates the Transformer and SSM layers under this unified positional encoding scheme. At a 4 sequenceK length, TransXSSM exhibits training and inference speeds that are 42.3% and 29.5% faster, respectively, relative to standard Transformer models. It also delivers higher accuracy: under comparable settings, it surpasses a Transformer baseline by over 4% on language modeling benchmarks.TransXSSM furthermore scales more effectively: TransXSSM-1.3B gains 7.22% in average accuracy over its 320M version (versus about 6% gains for equivalent Transformers or SSMs). Our results show that unified positional encoding resolves positional incompatibility in hybrid models, enabling efficient, high-performance long-context modeling.
BibTeX
@misc{wu2025transxssm,
  author = {Bingheng Wu and Jingze Shi and Yifan Wu and Nan Tang and Yuyu Luo},
  title = {{TransXSSM: A Hybrid Transformer State Space Model with Unified Rotary Position Embedding}},
  year = {2025},
  eprint = {2506.09507},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.09507},
}
Figure