Token Turing Machines are Efficient Vision Models

Purvish Jajal, Nick John Eliopoulos, Benjamin Shiue-Hal Chou, George K. Thiravathukal, James C. Davis, Yung-Hsiang Lu

TL;DR

This paper introduces Vision Token Turing Machines (ViTTM), a novel architecture that augments Vision Transformers (ViTs) with a memory mechanism inspired by Neural Turing Machines. ViTTMs achieve significantly lower latency and comparable or better accuracy on image classification and segmentation tasks by processing fewer "process tokens" while leveraging a "memory token" stream for information storage and retrieval.

What Problem Does This Paper Address?

Vision Transformers (ViTs) have revolutionized computer vision, but their computational demands, particularly the quadratic complexity with respect to input size, lead to high latency and significant computational costs. Existing methods to mitigate this, like token sparsification, often compromise accuracy. The authors aim to create a ViT architecture that is both efficient (low latency) and accurate, addressing the trade-off between computational cost and performance.

Methods

The core innovation is the Vision Token Turing Machine (ViTTM) architecture, a hybrid of ViT and memory-augmented neural networks. ViTTMs process an input image through two parallel streams:

  • Process Tokens: A smaller set of tokens that undergo the standard transformer encoder blocks. This stream is designed to be compute-heavy but process fewer tokens, thus reducing latency.
  • Memory Tokens: A larger set of tokens that act as an external memory. These tokens are read from and written to by the process tokens via specialized "read-write heads."

This memory mechanism allows the model to store and retrieve information, compensating for the reduced number of process tokens and maintaining accuracy. The architecture integrates these memory operations on a per-layer basis within the transformer blocks. The authors explore various design choices for initializing tokens, read-write heads, and fusion mechanisms, ultimately adopting Linear Attention for read-write heads and an "Add" fusion strategy.

Datasets

  • ImageNet-1K: Used for image classification tasks.
  • ADE20K: Used for semantic segmentation tasks.

The paper does not introduce new benchmarks but evaluates its model against existing state-of-the-art methods on these standard datasets.

Key Insights and Ideas

  • Memory as a Latency Reducer: The key insight is that by using a memory mechanism, ViTs can afford to process fewer "active" tokens (process tokens) without sacrificing accuracy. The memory tokens act as a compressed representation or cache of information, allowing the model to retain context and detail.
  • Two-Stream Efficiency: The parallel processing of a small set of compute-heavy process tokens and a larger set of memory tokens is crucial for achieving both speed and accuracy. This contrasts with methods that simply reduce token count without a mechanism to retain information.
  • Per-Layer Memory Integration: Integrating memory operations directly within each transformer layer, rather than as a separate module, allows for more dynamic and fine-grained information exchange, contributing to the model's efficiency and effectiveness.
  • Linear Attention for Efficiency: The choice of Linear Attention for read-write heads is highlighted as a critical factor for efficiency, offering comparable accuracy to Cross Attention but with significantly lower computational complexity.

Key Results

  • ImageNet-1K Classification: ViTTM-B achieved 82.9% accuracy, a 1.9% improvement over ViT-B (81.0%), while being 56% faster (234.1ms vs. 529.5ms) and using 2.4x fewer FLOPs.
  • ADE20K Semantic Segmentation: ViTTM-B achieved 45.17 mIoU with 26.8 FPS on an NVIDIA A30 GPU, a 94% FPS improvement over ViT-B (13.8 FPS) with a comparable mIoU (45.65 mIoU for ViT-B).
  • Ablation Studies: Showed that memory tokens generally improve accuracy, though the marginal benefit of adding more memory tokens is small. Conversely, increasing process tokens has a larger impact on accuracy. The choice of read-write head and fusion mechanism significantly impacts performance and efficiency.

Limitations and Future Work

  • Training Instabilities: The authors note that two-stream architectures can sometimes suffer from training instabilities, suggesting that a rigorous study of architectural choices and training dynamics is needed.
  • Memory Token Initialization: While the paper focuses on initializing process tokens, the initialization of memory tokens is not explored. The authors suggest this could be a direction for multi-modal applications.
  • Further Latency Optimization: Future work could focus on optimizing the latency of read-write mechanisms, such as exploring different fusion operators.
  • Multi-Modal Applications: The memory stream could potentially be initialized using information from other modalities, opening avenues for multi-modal ViTTM models.

Summary updated

Abstract

We propose Vision Token Turing Machines (ViTTM), an efficient, low-latency, memory-augmented Vision Transformer (ViT). Our approach builds on Neural Turing Machines (NTM) and Token Turing Machines (TTM), which were applied to NLP and sequential visual understanding tasks. ViTTMs are designed for non-sequential computer vision tasks such as image classification and segmentation. Our model creates two sets of tokens: process tokens and memory tokens; process tokens pass through encoder blocks and read-write from memory tokens at each encoder block in the network, allowing them to store and retrieve information from memory. By ensuring that there are fewer process tokens, we are able to reduce the inference time of the network while maintaining its accuracy. On ImageNet-1K, the state-of-the-art ViT-B has median latency of 529.5ms and 81.0% accuracy, while our ViTTM-B is 56% faster (234.1ms), with 2.4x fewer FLOPs, with an accuracy of 82.9%. On ADE20K semantic segmentation, ViT-B achieves 45.65mIoU at 13.8 frame-per-second (FPS) whereas our ViTTM-B model acheives a 45.17 mIoU with 26.8 FPS (+94%).
BibTeX
@inproceedings{jajal2025token,
  author = {Purvish Jajal and Nick John Eliopoulos and Benjamin Shiue-Hal Chou and George K. Thiravathukal and James C. Davis and Yung-Hsiang Lu},
  title = {{Token Turing Machines are Efficient Vision Models}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00767},
}
Figure