Token Turing Machines are Efficient Vision Models

George K. Thiruvathukal, PhD, Purvish Jajal, Nick John Eliopoulos, Benjamin Shiue-Hal Chou, George K. Thiravathukal, James C. Davis, Yung-Hsiang Lu

Introduction

Vision Transformers (ViTs) have revolutionized computer vision, yet they face a significant hurdle: computational complexity. The self-attention mechanism's cost scales quadratically with the number of input tokens, making high-resolution processing prohibitively expensive. While "token sparsification" (pruning less important patches) offers a solution, it often sacrifices global context and accuracy.

Vision Token Turing Machines (ViTTM) address this by decoupling heavy computation from information storage. Inspired by Neural Turing Machines (NTMs), ViTTM introduces a dual-token architecture that maintains a full representation of the input while restricting expensive operations to a small subset of "process tokens."

Architectural Innovation: Process and Memory Tokens

The core of ViTTM is the separation of tokens into two distinct streams:

  1. Process Tokens ($P$): A small, fixed set of tokens that undergo standard Transformer encoder processing.
  2. Memory Tokens ($M$): A larger set of tokens acting as a persistent storage bank, bypassing the expensive encoder blocks.
Conceptual comparison of NTM/TTM, standard ViT, and the proposed ViTTM architecture.

At each layer, the model executes a Read-Process-Write cycle. Process tokens use efficient Linear Attention to "read" from memory, are processed by a standard encoder block, and then "write" updated information back to the memory bank. This $O(N)$ complexity for memory management ensures that the system remains efficient even with large memory banks.

Performance and Efficiency

ViTTM significantly pushes the Pareto frontier for vision models. On the ImageNet-1K benchmark, ViTTM-B achieves 82.9% accuracy with a median latency of 234.1ms. This represents a 56% speedup over the standard ViT-B (81.0% accuracy, 529.5ms) while simultaneously improving classification performance.

ImageNet-1K Accuracy vs. Latency (ms). ViTTM models (red stars) demonstrate a superior trade-off compared to standard ViT and other efficient variants.

The efficiency gains extend to dense prediction tasks. In semantic segmentation on ADE20K, ViTTM-B reached 26.8 FPS—a 94% increase in throughput over the baseline ViT-B—with negligible loss in mIoU (45.17 vs 45.65).

Key Insights

  • Linear Attention Efficiency: Linear Attention serves as an effective, low-overhead gateway for memory access, matching the performance of standard cross-attention at a fraction of the cost.
  • Storage vs. Computation: Experiments revealed that the memory stream's primary value lies in information retention rather than active computation; adding complex layers to the memory stream yielded diminishing returns.
  • Asymmetric Patching: A simple yet effective strategy for initializing the two streams is using different patch sizes—larger for process tokens and smaller for memory—to balance global context with local detail.

Conclusion and Future Directions

ViTTM demonstrates that vision models do not need to "think" about every pixel at every layer. By utilizing a memory-augmented architecture, ViTTM achieves state-of-the-art efficiency. Future work may explore multi-modal memory initialization (e.g., text or audio) and hardware-level optimizations through custom fused kernels to further enhance performance.

Summary updated

Abstract

We propose Vision Token Turing Machines (ViTTM), an efficient, low-latency, memory-augmented Vision Transformer (ViT). Our approach builds on Neural Turing Machines and Token Turing Machines, which were applied to NLP and sequential visual understanding tasks. ViTTMs are designed for non-sequential computer vision tasks such as image classification and segmentation. Our model creates two sets of tokens: process tokens and memory tokens; process tokens pass through encoder blocks and read-write from memory tokens at each encoder block in the network, allowing them to store and retrieve information from memory. By ensuring that there are fewer process tokens than memory tokens, we are able to reduce the inference time of the network while maintaining its accuracy. On ImageNet-1K, the state-of-the-art ViT-B has median latency of 529.5ms and 81.0% accuracy, while our ViTTM-B is 56% faster (234.1ms), with 2.4 times fewer FLOPs, with an accuracy of 82.9%. On ADE20K semantic segmentation, ViT-B achieves 45.65mIoU at 13.8 frame-per-second (FPS) whereas our ViTTM-B model acheives a 45.17 mIoU with 26.8 FPS (+94%).
BibTeX
@inproceedings{phd2025token,
  author = {George K. Thiruvathukal, PhD and Purvish Jajal and Nick John Eliopoulos and Benjamin Shiue-Hal Chou and George K. Thiravathukal and James C. Davis and Yung-Hsiang Lu},
  title = {{Token Turing Machines are Efficient Vision Models}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00767},
}
Figure