Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architectures

Georgios Pantazopoulos, Malvina Nikandrou, Ioannis Konstas, Alessandro Suglia

TL;DR

This paper investigates whether hybrid architectures combining Transformers and State Space Models (SSMs) can deliver the "best of both worlds": the linear-time efficiency of SSMs and the precise in-context retrieval capabilities of Transformers. Through controlled synthetic tasks, the authors show that hybrid models excel at content-based retrieval and length extrapolation, but Transformers maintain a clear edge in position-based tracking. Crucially, the study reveals that SSMs naturally develop unique, "locality-aware" spiral representations in their embedding space, which fundamentally alters how they process sequence structure compared to attention mechanisms.

What Problem Does This Paper Address?

While Transformers dominate sequence modeling, their self-attention mechanism scales quadratically ($O(L^2)$) with sequence length, creating massive computational and memory bottlenecks for long contexts. State Space Models (SSMs), such as Mamba, offer a linear-time ($O(L)$) alternative with constant memory during inference. However, pure SSMs struggle with precise in-context retrieval—the ability to locate and extract specific tokens from a massive prompt.

To solve this, researchers have begun blending the two into hybrid architectures. Yet, exactly how these hybrids perform on retrieval tasks, how data-efficient they are, and how they represent positional information internally has remained a black box. This paper pulls back the curtain on these dynamics.

Methods

The authors evaluate three model families—Transformers, SSMs (Mamba and Mamba2), and Hybrids—across two distinct synthetic tasks designed to isolate architectural traits from semantic knowledge:

  1. N-gram Retrieval: The model must find a specific query $n$-gram within a sequence and output the $k$ tokens that immediately follow it. This tests content-based associative recall.
  2. Position Retrieval: A "two-hop" task where the model is given a query token, must locate it in the sequence, and then output its exact numerical positional index (represented as distinct coordinate tokens).

The authors test two hybrid strategies:

  • Interleaved (Hybrid_I): A Transformer block is inserted after every $N$ SSM blocks to "correct" the compressed hidden state.
  • Two-Stream (Hybrid_2S): Transformer and SSM blocks process inputs in parallel, and their outputs are fused via a learnable tanh gating mechanism.
Figure 2: Illustration of hybrid architectures. Left: Interleaved Mamba and Transformer blocks. A Trans- former block is inserted every N Mamba blocks. Right: Two stream block with a gate mechanism. The outputs from both streams are fused with a learnable gating mechanism.

Datasets

The evaluation relies entirely on synthetic datasets to strip away semantic biases:

  • For N-gram retrieval, sequences use a vocabulary size of 30, with queries placed either at the end (suffix variant) or the beginning (prefix variant).
  • For Position retrieval, sequences consist of 200 randomly shuffled unique tokens paired with 200 distinct positional coordinate tokens.
Examples of training sequences for the n-gram and position retrieval tasks.

Key Insights and Ideas

  • The SSM "Spiral" Representation: Through Principal Component Analysis (PCA), the authors discovered that SSM-based models naturally organize positional tokens into a continuous, two-dimensional spiral trajectory. Tokens representing adjacent positions become geometric neighbors in the embedding space. Transformers show no such structure.
  • The Serial Position Effect: SSMs and hybrids exhibit a human-like learning bias, mastering retrieval for tokens at the very beginning and very end of a sequence first, before gradually filling in the middle. Transformers, by contrast, learn the entire sequence uniformly and almost instantaneously once they hit a training threshold.
  • Attention as a State Modifier: In hybrid models, even a low frequency of Transformer blocks acts as a powerful corrector, editing the compressed hidden state of the SSM to restore discarded details necessary for retrieval.

Key Results

  • Hybrids Win on Content Retrieval: For the $n$-gram retrieval task, hybrid models (especially interleaved ones) converge significantly faster than pure SSMs, requiring an order of magnitude less data to reach $\ge95\%$ accuracy. They also demonstrate near-perfect length generalization, extrapolating smoothly to sequences 10x longer than those seen during training.
Figure 3: (a) N-gram retrieval data efficiency. We train models to retrieve a sequence of k = 3 tokens that follow a randomly selected n-gram (n = 2) in a string of length ≤100, and evaluate on strings of length 100. While Transformers train significantly faster than SSMs, hybrid architectures are converging even faster. (b) Effect of state space dimension. We train SSM models with different state space dimensions. Both Mamba versions benefit from a higher capacity state space, their performance is still inferior to that of a standard Transformer, shown here as a violet dashed line. (c) Effect of interleaved SSM/Transformer blocks in hybrid-interleaved models (HybridI). We explore the effect of a Transformer block after N = {1, 2, 3, 4} SSM blocks. A single Transformer layer complements the SSM stack by correcting the hidden state and yielding increased performance than a pure SSM model shown in green. Models with interleaving Transformer blocks after N < 4 SSM blocks even surpass the performance of a pure Transformer.
  • Transformers Dominate Position Tracking: On the two-hop position retrieval task, pure Transformers with Rotary Position Embeddings (RoPE) converge the fastest. Standalone SSMs completely fail to solve this task within the 20-million-example training budget. The authors attribute this to the SSMs' locality-aware embeddings, which inadvertently act as a bottleneck for non-linear, coordinate-based lookups.

Limitations and Future Work

The primary limitation is the reliance on synthetic tasks with controlled complexity. While these tasks serve as excellent proxies for architectural mechanics, they do not fully capture the nuances of natural language modeling or multimodal applications. Additionally, this study focuses exclusively on decoder-only architectures; exploring how SSM-attention hybrids behave in encoder-decoder or encoder-only configurations remains an open avenue for future research.

Summary updated

Abstract

Transformers excel at in-context retrieval but suffer from quadratic complexity with sequence length, while State Space Models (SSMs) offer efficient linear-time processing but have limited retrieval capabilities. We investigate whether hybrid architectures combining Transformers and SSMs can achieve the best of both worlds on two synthetic in-context retrieval tasks. The first task, n-gram retrieval, requires the model to reproduce an n-gram that succeeds the query within the input sequence. The second task, position retrieval, presents the model with a query token and requires it to perform a two-hop lookup: first locating the corresponding element in the sequence, and then outputting its positional index. Under controlled conditions, we assess data efficiency, length generalization, robustness to out of domain training examples, and learned representations across Transformers, SSMs, and hybrid architectures. We find that hybrid models outperform SSMs and match or exceed Transformers in terms of data efficiency and extrapolation for tasks that require precise information retrieval from the input context. However, Transformers maintain superiority in position retrieval tasks. Through representation analysis, we discover that SSM-based models develop locality-aware embeddings where tokens representing adjacent positions become neighbors in embedding space, forming interpretable structures. This property is absent in Transformers as causal attention is sufficient for acquiring positional associations, and the introduction of positional encoding amplifies this behavior, leading to improvement in data efficiency. SSMs on the other hand update their internal representations incrementally and without positional encodings, are required to learn these associations. Our findings reveal fundamental differences in how Transformers and SSMs, and hybrid models learn positional associations.
BibTeX
@article{pantazopoulos2026retrievit,
  author = {Georgios Pantazopoulos and Malvina Nikandrou and Ioannis Konstas and Alessandro Suglia},
  title = {{Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architectures}},
  journal = {Trans. Mach. Learn. Res.},
  year = {2026},
}
Figure