Improved Image Classification with Token Fusion

Keong Hun Choi, Jin Woo Kim, Yao Wang, Jong Eun Ha

TL;DR

This paper introduces "Token Fusion," a hybrid architecture that combines the local feature extraction strengths of CNNs with the global modeling power of Vision Transformers (ViTs). By treating CNN feature map pixels as specialized tokens and integrating them through a layer-by-layer "Mixing Block," the authors achieve state-of-the-art accuracy on ImageNet-1K while using 36% fewer parameters than leading competitors.

What Problem Does This Paper Address?

In the world of computer vision, we have two dominant paradigms: Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs).

  • CNNs are masters of locality; their sliding windows are perfect for picking up fine-grained textures and edges, but they struggle to understand how a pixel in the top-left corner relates to one in the bottom-right (global context).
  • Transformers, conversely, use self-attention to look at the whole image at once. While this provides excellent global context, it is computationally expensive and often misses the "local" nuances unless the model is massive or trained on gargantuan datasets.

The authors tackle the challenge of creating a unified architecture that captures both local and global information efficiently, without the massive memory overhead typically required by high-resolution Transformers.

Methods

The core innovation is the treatment of CNN outputs as "tokens." In a standard ViT, an image is chopped into patches, and each patch becomes a token. In this paper, the authors take a pre-trained ResNet-101 and treat the pixels of its feature maps as a second type of token.

They proposed and compared three fusion strategies:

  1. Late Token Fusion (Parallel): CNN and Transformer branches process data separately in parallel, and their outputs are concatenated only at the very end.
  2. Early Token Fusion: CNN features are extracted from the original image and merged with the image patches before entering the Transformer encoder.
  3. Layer-by-Layer Fusion (The Winner): This approach uses a "Mixing Block" where CNN and Transformer modules are interleaved. A total of five mixing blocks correspond to the hierarchical stages of the ResNet backbone, allowing the model to refine local and global features simultaneously throughout the network.

Additionally, the authors moved away from the traditional "Class Token" ([CLS]) used in ViTs. Instead, they experimented with different pooling heads, finding that channel-wise pooling provided the best performance for their hybrid tokens.

Datasets

The primary benchmark used for training and evaluation was ImageNet-1K (approx. 1.2 million images). To test the model's ability to scale, they also evaluated it using ImageNet-22K (a much larger dataset with ~14 million images). They utilized a pre-trained ResNet-101 as the convolutional backbone for their experiments.

Key Insights and Ideas

  • CNN Pixels as "Pre-Digested" Tokens: By using CNN feature maps as tokens, the Transformer doesn't have to learn basic textures from scratch. It receives tokens that already contain rich local spatial information.
  • Deep Integration Beats Shallow Fusion: The "Layer-by-Layer" approach significantly outperformed Early or Late fusion. This suggests that the interaction between local and global features needs to happen at multiple levels of abstraction (from low-level edges to high-level concepts).
  • Rethinking the Classification Head: The standard ViT approach of using a single [CLS] token for classification was less effective than pooling across the rich set of hybrid tokens. Channel-wise pooling, in particular, allowed the model to better leverage the diverse information present in the fused feature set.
  • Parameter Efficiency: One of the most striking findings was that their model achieved 87.8% accuracy on ImageNet-1K with 140M parameters, outperforming the MViTv2-L model which requires 218M parameters.

Key Results

  • Top Performance: The Layer-by-Layer model with channel-wise pooling achieved a top-1 accuracy of 87.77% on ImageNet-1K.
  • Efficiency Gain: The proposed method provides state-of-the-art results while using 36% fewer parameters than the second-best algorithm (MViTv2-L).
  • Scaling: When trained on ImageNet-22K, the model reached 87.95% accuracy. While this is a strong result, it showed a smaller "jump" in performance compared to some other models when moving to larger datasets, indicating a potential area for further optimization.

Limitations and Future Work

The authors acknowledge a few areas for improvement:

  • Data Scaling Diminishing Returns: While the model is highly efficient on ImageNet-1K, it didn't show the same massive performance boost on ImageNet-22K that some pure Transformer models exhibit. The authors suggest this requires further investigation into how hybrid tokens behave with massive data.
  • Task Expansion: This paper focused strictly on image classification. The authors intend to extend this "Token Fusion" concept to more complex tasks like object detection and semantic segmentation, where the balance of local precision and global context is even more critical.

Summary updated

Abstract

In this paper, we propose a method using the fusion of CNN and transformer structure to improve image classification performance. In the case of CNN, information about a local area on an image can be extracted well, but there is a limit to the extraction of global information. On the other hand, the transformer has an advantage in relatively global extraction, but has a disadvantage in that it requires a lot of memory for local feature value extraction. In the case of an image, it is converted into a feature map through CNN, and each feature map's pixel is considered a token. At the same time, the image is divided into patch areas and then fused with the transformer method that views them as tokens. For the fusion of tokens with two different characteristics, we propose three methods: (1) late token fusion with parallel structure, (2) early token fusion, (3) token fusion in a layer by layer. In an experiment using ImageNet 1k, the proposed method shows the best classification performance.
BibTeX
@inproceedings{choi2022improved,
  author = {Keong Hun Choi and Jin Woo Kim and Yao Wang and Jong Eun Ha},
  title = {{Improved Image Classification with Token Fusion}},
  booktitle = {IEEE Access},
  year = {2022},
}
Figure