Abstract
Standard Transformer architectures are limited by the quadratic computational complexity of self-attention and a focus on intra-sample relationships. We present the Cached Transformer, which introduces a Gated Recurrent Cache (GRC) to maintain a differentiable memory of historical representations. By employing GRC-Attention, the model effectively captures long-range dependencies and inter-sample prototypical features. Our approach demonstrates significant performance improvements across computer vision and natural language processing tasks, offering a superior accuracy-to-efficiency trade-off compared to scaling model depth or width.
Introduction
The self-attention mechanism, while powerful, scales poorly with sequence length ($O(T^2)$), hindering the modeling of long-range dependencies. Furthermore, in domains like computer vision, Transformers typically process images in isolation, failing to exploit shared "prototypical" knowledge across different samples. Existing memory-based solutions often rely on simple, non-differentiable buffers (e.g., FIFO) that lack the capacity to selectively retain relevant information and are memory-intensive.
The Gated Recurrent Cache (GRC)
The core innovation of the Cached Transformer is the Gated Recurrent Cache (GRC). This module acts as a meta-learner that compresses historical data into fixed-length embedding vectors, allowing the model to traverse "infinite" history in constant time.
GRC-Attention Mechanism
The model replaces or augments standard self-attention with GRC-Attention. In this setup, the query attends to both the current input tokens ($X_t$) and the memory cache ($C_t$). The final output is a learnable interpolation between the standard self-attention result and the cached attention result, controlled by a gating parameter $\sigma(\lambda)$.
Differentiable Gating
Inspired by Gated Recurrent Units (GRUs), the GRC utilizes update gates ($g_u$) and reset gates ($g_r$). These gates allow the model to dynamically determine which historical information to preserve and how to integrate new inputs. Unlike static buffers, this cache is updated during the forward pass and optimized via backpropagation, enabling the model to learn an optimal representation of "memory" for the task at hand.
Experimental Evaluation
The Cached Transformer was evaluated across computer vision (ImageNet, COCO), language modeling (WikiText-103), and long-range modeling (LRA) benchmarks.
Performance Gains in Vision
On ImageNet-1k, the integration of GRC into the PVT-Tiny backbone resulted in a Top-1 accuracy increase of 3.3% (75.1% to 78.4%). On COCO object detection, the PVT-Medium backbone saw a massive +4.6 box AP boost. Notably, a GRC-enhanced PVT-Small outperforms a vanilla PVT-Medium while remaining more efficient.
Language and Long-Range Modeling
In language modeling on WikiText-103, the GRC-cached model surpassed Transformer-XL by 1.1 perplexity points, despite using a more compact memory representation. In the Long Range Arena's "Long ListOps" task, the GRC improved the accuracy of BigBird and Reformer by 1.39% and 0.58%, respectively, proving its efficacy in handling sequences up to 2,000 tokens.
Discussion and Insights
Analysis of attention maps indicates a functional division: standard self-attention focuses on instance-specific details, while cached attention captures instance-invariant prototypical features. This dual-pathway approach acts as a regularizer, improving generalization. Furthermore, the authors observed a coarse-to-fine processing pattern, where earlier layers rely more heavily on the cache for coarse instance distinction, while later layers shift toward self-attention for fine-grained decision-making.
While GRC-Attention adds roughly 10-15% more parameters and FLOPs, and results in a 15-20% drop in throughput, the performance gains are far superior to simply making baseline models deeper or wider.
Conclusion
The Cached Transformer provides a robust, plug-and-play framework for enhancing Transformers with differentiable memory. By efficiently capturing both long-range and cross-sample dependencies, it sets new performance standards across multiple modalities while maintaining a competitive computational profile.
Summary updated