TL;DR
Vision Token Turing Machines (ViTTM) introduce an efficient, memory-augmented architecture for Vision Transformers (ViTs) by decoupling computation from data storage. By routing a small set of "process tokens" through heavy transformer blocks while interacting with a larger, lightweight set of "memory tokens" via efficient read-write heads, ViTTM slashes latency by up to 56% and FLOPs by 2.4× while actually improving accuracy on core vision tasks.
What Problem Does This Paper Address?
Vision Transformers (ViTs) have become the bedrock of modern computer vision, powering everything from image classification to semantic segmentation. However, their success comes at a steep computational cost: the self-attention mechanism scales quadratically ($O(N^2)$) with the number of input tokens (patches).
To make ViTs viable for latency-sensitive or resource-constrained applications, researchers often turn to token sparsification—pruning or merging tokens during inference. While this speeds up the network, it frequently degrades accuracy because discarded tokens might contain fine-grained, contextual information that the model needs later. The authors of this paper ask: Can we drastically reduce the number of active tokens being processed without losing the rich information contained in the full image?
Methods
The authors introduce Vision Token Turing Machines (ViTTM), an architecture inspired by Neural Turing Machines (NTMs) but adapted for non-sequential vision tasks.
Instead of processing a single uniform stream of tokens, ViTTM splits the visual data into two distinct streams:
- Process Tokens ($P$): A small, compute-heavy set of tokens generated using a larger patch size. These are the only tokens that pass through the standard, expensive ViT Encoder Blocks.
- Memory Tokens ($M$): A larger, compute-light set of tokens generated using a smaller patch size. These tokens act as an external storage bank, holding rich spatial details.
At each layer of the network, the model performs a three-step sequence:
- Read: Process tokens use an ultra-efficient Linear Attention mechanism to extract relevant context from the memory tokens.
- Process: The read information is fused into the process tokens (using simple element-wise addition), which then pass through a standard transformer encoder block.
- Write: The updated process tokens write new information back into the memory tokens via another linear attention operation.
Crucially, unlike competing two-stream architectures (such as LookupViT), the memory tokens themselves never pass through Multi-Layer Perceptrons (MLPs) or self-attention blocks. They are purely read from and written to, keeping the memory overhead incredibly lightweight.
Datasets
The model was evaluated on standard, widely accepted vision benchmarks:
- ImageNet-21K: Used for large-scale pre-training.
- ImageNet-1K: Used for fine-tuning and evaluating image classification accuracy.
- ADE20K: Used to evaluate performance on dense prediction via semantic segmentation (leveraging the Segmenter framework).
Key Insights and Ideas
- Linear Attention is the Enabler: Standard cross-attention depends on the length of both token sequences, which would negate the savings of reducing process tokens. By utilizing Linear Attention, the computational complexity of the read/write heads depends on the length of only one sequence, making the memory access incredibly cheap.
- Memory Refinement is Overkill: A major "aha" moment from their ablations showed that adding an MLP to process or "refine" memory tokens yielded virtually zero accuracy gains while significantly increasing FLOPs and latency. Pure read/write storage is sufficient.
- Asymmetric Token Roles: Increasing the number of process tokens drastically improves accuracy (as they handle the heavy representation learning), whereas increasing memory tokens yields diminishing returns after a certain point. This justifies their design of keeping process tokens minimal and memory tokens dense.
Key Results
ViTTM expands the Pareto front for the accuracy-latency trade-off, outperforming standard ViTs and contemporary two-stream models.
Image Classification (ImageNet-1K)
- ViTTM-B(64,64) achieved 82.9% Top-1 accuracy with a median latency of 234.1 ms.
- Compared to the baseline ViT-B, ViTTM-B is 56% faster (down from 529.5 ms), uses 2.4× fewer FLOPs (7.08G vs 16.87G), and delivers a +1.9% accuracy boost.
- Compared to its closest competitor, LookupViT, ViTTM achieves either significantly lower latency at comparable accuracy or much higher accuracy at equivalent latencies.
Semantic Segmentation (ADE20K)
- ViTTM-B(64,64) matched the mIoU of a standard ViT-B baseline (~45.17 vs 45.65) while boosting throughput significantly.
- It achieved a 94% increase in Frames Per Second (FPS) on an NVIDIA A30 GPU (26.8 FPS vs 13.8 FPS) and a 37% increase on an NVIDIA A100 GPU.
Limitations and Future Work
While ViTTM shows impressive hardware efficiencies, the authors note a few areas for improvement:
- Training Instability: Not refining the memory stream can make training unstable. While layer normalization helps, the authors had to rely on large batch sizes (up to 4096 during pre-training) to ensure smooth convergence.
- Unexplored Hyperparameters: The architecture leaves several design variables unexamined, such as using different embedding dimensions for the process and memory streams, or varying the network depth.
- Multi-Modal Potential: Because the memory stream is decoupled from the processing stream, future work could explore initializing the memory tokens with an entirely different modality (e.g., text or audio), opening the door for efficient multi-modal transformers.
Summary updated