Introduction
Standard Transformers have revolutionized deep learning, yet they face a fundamental bottleneck: the computational complexity of self-attention grows quadratically with sequence length ($O(T^2)$). This scaling issue makes it prohibitively expensive to process long-range dependencies or maintain a persistent memory of historical data. While existing solutions like Transformer-XL attempt to mitigate this by caching previous tokens, they typically rely on a simple First-In-First-Out (FIFO) approach. This treats all past information as equally relevant, failing to distinguish between critical context and noise.
The Cached Transformer addresses these limitations by introducing a differentiable Gated Recurrent Cache (GRC). This mechanism allows models to adaptively learn what to store and what to discard, effectively compressing historical data into a fixed-length memory that can be queried in constant time.
The Gated Recurrent Cache (GRC)
The core innovation of this work is the GRC-Attention mechanism, which augments standard self-attention with a learnable memory module. Instead of attending solely to the current input tokens $X_t$, the model maintains a continuous cache $C_t$.
GRC-Attention Mechanism
As shown in the architecture below, the attention process is bifurcated. The model performs standard self-attention on the current tokens to capture local context, while simultaneously performing "cached attention" over the memory module to retrieve historical or global information. These two streams are merged using a learnable, head-wise ratio $\sigma(\lambda_h)$, allowing the model to dynamically balance current perception with stored knowledge.
Gating and Differentiable Updates
Inspired by Gated Recurrent Units (GRUs), the GRC utilizes a gating mechanism to manage memory state transitions:
- Reset Gate ($g_r$): Determines which components of the previous cache $C_{t-1}$ are no longer relevant to the current task.
- Update Gate ($g_u$): Controls the extent to which new input features from $X_t$ are integrated into the updated cache $C_t$.
Because these gates are fully differentiable, the model learns the optimal caching strategy directly from the data during training. In vision tasks, this cache acts as a repository for prototypical feature representations across different images, facilitating a form of "global" cross-sample knowledge.
Experimental Evaluation
The authors evaluated the Cached Transformer across a wide array of benchmarks, including ImageNet-1k for vision, COCO for object detection, and WikiText-103 for language modeling.
Key Results in Computer Vision
Integrating GRC into existing vision backbones yielded substantial performance gains without the massive parameter increases typically required for such jumps. For instance, adding GRC to a standard ViT-Small improved ImageNet top-1 accuracy by 1.4%. The impact was even more pronounced on the Pyramid Vision Transformer (PVT-Tiny), which saw a 3.3% accuracy boost.
Beyond classification, the GRC-enhanced PVT-Tiny backbone achieved a massive 4.3 point increase in box AP for object detection on the COCO dataset. These results demonstrate that the differentiable cache provides a more efficient path to high performance than simply scaling model depth or width.
Language and Long-Range Tasks
On the WikiText-103 language modeling benchmark, the Cached Transformer outperformed the popular Transformer-XL by 1.1 perplexity points. This confirms that the gating mechanism is significantly more effective at capturing long-term linguistic dependencies than simple FIFO caching. Furthermore, in the Long Range Arena (LRA) "Long ListOps" task, a GRC-enhanced BigBird model outperformed its standard counterpart by 1.39%, proving its utility for sequences up to 2,000 tokens.
Discussion and Insights
One of the most compelling findings is the specialization of the two attention branches. Visualization of the attention maps reveals that the cache tends to focus on "instance-invariant" features—the general, public characteristics of a category. In contrast, the self-attention branch focuses on "instance-specific" details, such as unique textures or private features of a specific object.
Additionally, the model exhibits a strategic use of memory across its depth. It relies heavily on the cache in the early and middle layers to build a contextual foundation, then shifts toward standard self-attention in the final layers to refine its specific predictions.
Conclusion
The Cached Transformer offers a scalable and efficient solution to the memory limitations of standard self-attention. While the GRC module introduces a slight computational overhead (roughly 10-15% more FLOPs), the resulting accuracy and efficiency gains—particularly on long-sequence and high-resolution vision tasks—are substantial. Future work may explore extending this differentiable cache to enable "infinite" context windows in large language models (LLMs) or applying it to complex multi-modal reasoning tasks.
Summary updated