LLaMA Decoder As Vision Transformer
Lists
Only you see this
Abstract
Using the same architecture for text and image is important for AI standardization. Recent multimodal models use a decoder-only Transformer to generate text and an encoder-only Transformer to extract image features. Can images use exactly the same language architecture? To answer this question, we aim at a LLaMa decoder as vision Transformer (ViT) classifier in this paper. Specifically, we start our trajectory by “LLaMAfy” a standard ViT step-by-step, i.e., feed-forward net, normalization layer, causal self-attention and positional embedding, and point out a key issue—attention collapse—that result in the failure to the network training. Motivated by this observation, we propose post-sequence class token, enabling causal self-attention to efficiently capture the entire image’s information. To improve model optimization behavior and enhance performance, we then introduce a soft mask strategy to gradually transform the attention from bi-directional to causal mode. The tailored model, dubbed as image LLaMA (iLLaMA), maintains high consistency with LLaMA architecture, while matching up well against ViT, achieving 75.1% ImageNet top-1 accuracy with only 5.7M parameters. Scaling the model to ∼310M and pre-training on ImageNet-21K further enhances the accuracy to 86.0%. Its causal self-attention boosts computational efficiency and learns complex representation by elevating attention map ranks. Extensive experiments demonstrate iLLaMA’s reliable properties: shape-texture bias, calibration, quantization compatibility, ADE20K segmentation and CIFAR transfer learning. We hope our study can kindle fresh views to visual architectures in the era of LLMs and contributes to standardized AI models.
BibTeX
@misc{wang2025llama,
author = {Jiahao Wang and Wenqi Shao and Mengzhao Chen and Chengyue Wu and Yong Liu and Taiqiang Wu and Kaipeng Zhang and Songyang Zhang and Kai Chen and Ping Luo},
title = {{LLaMA Decoder As Vision Transformer}},
year = {2025},
url = {https://openreview.net/forum?id=GraOHDxFjl},
howpublished = {OpenReview},
note = {Submitted to ICLR 2025},
}