Cached Transformers: Improving Transformers with Differentiable Memory Cachde

Zhaoyang Zhang, Wenqi Shao, Yixiao Ge, Xiaogang Wang, Jinwei Gu, Ping Luo

Abstract

This work introduces a new Transformer model called Cached Transformer, which uses Gated Recurrent Cached (GRC) attention to extend the self-attention mechanism with a differentiable memory cache of tokens. GRC attention enables attending to both past and current tokens, increasing the receptive field of attention and allowing for exploring long-range dependencies. By utilizing a recurrent gating unit to continuously update the cache, our model achieves significant advancements in \textbf{six} language and vision tasks, including language modeling, machine translation, ListOPs, image classification, object detection, and instance segmentation. Furthermore, our approach surpasses previous memory-based techniques in tasks such as language modeling and displays the ability to be applied to a broader range of situations.
BibTeX
@inproceedings{zhang2024cached,
  author = {Zhaoyang Zhang and Wenqi Shao and Yixiao Ge and Xiaogang Wang and Jinwei Gu and Ping Luo},
  title = {{Cached Transformers: Improving Transformers with Differentiable Memory Cachde}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i15.29636},
}
Figure