Factorization Vision Transformer: Modeling Long-Range Dependency With Local Window Cost

Haolin Qin, Daquan Zhou, Tingfa Xu, Ziyang Bian, Jianan Li

TL;DR

This paper introduces FaSA, a novel self-attention mechanism that factorizes the attention matrix to capture long-range dependencies at a computational cost equivalent to local window-based attention. This allows their proposed FaViT model to achieve state-of-the-art performance and robustness in computer vision tasks while being significantly more efficient than existing transformers.

What Problem Does This Paper Address?

Vision Transformers (ViTs) have shown remarkable success in computer vision, largely due to their ability to model long-range dependencies. However, their self-attention mechanism has a quadratic computational complexity with respect to image resolution, making them impractical for high-resolution images or resource-constrained devices. While methods like Swin Transformer reduce computational cost by using local windows, this comes at the expense of hindering long-range dependency modeling and potentially impacting model robustness. This paper aims to bridge this gap by developing a method that offers the best of both worlds: the efficiency of local windows and the powerful long-range modeling capabilities of global attention.

Methods

The core innovation is the Factorization Self-Attention (FaSA) mechanism. Instead of computing a full attention matrix, FaSA factorizes it into multiple sparse sub-attention matrices. This is achieved through a multi-step process:

  1. Head Grouping: The input feature map is divided into several groups along the channel dimension.
  2. Local Windowing: Each group's feature map is divided into non-overlapping local windows.
  3. Dilated Sampling: Within each local window, a fixed number of points are sampled using dilated sampling. The dilation rate increases across head groups, allowing for a coarser capture of information at higher dilation rates.
  4. Cross-Window Fusion: Features from sampled points at the same relative positions across different windows are fused. This step is crucial for aggregating long-range information.
  5. Mixed-grained Multi-head Attention: By progressively increasing local window sizes and dynamically adjusting dilation rates across different attention heads, FaSA captures information at various granularities, from fine-grained local details to broader contextual information.

This factorized approach allows FaSA to approximate the full attention matrix's representational capacity while maintaining linear computational complexity with respect to image resolution. The Factorization Vision Transformer (FaViT) is built upon this FaSA mechanism, featuring a hierarchical structure with multiple stages.

Datasets

The paper uses standard datasets for evaluation:

  • ImageNet-1K: For image classification.
  • ImageNet-C: For evaluating robustness against common image corruptions (blur, noise, digital, weather).
  • Clothing1M & WebVision: For evaluating robustness against label noise.
  • iNaturalist 2018: For evaluating robustness against class imbalance.
  • COCO 2017: For object detection and instance segmentation.
  • ADE20K: For semantic segmentation.
  • CIFAR100: Used for ablation studies.

Key Insights and Ideas

  • Factorization as a Trade-off Solution: The paper demonstrates that factorizing the attention matrix is a viable strategy to achieve both efficiency and long-range dependency modeling, overcoming the inherent trade-off in previous methods.
  • Dilated Sampling for Mixed Granularity: The intelligent use of dilated sampling with increasing dilation rates across attention heads is key to capturing information at multiple scales, preventing the loss of fine-grained details that might occur with simpler local attention.
  • Cross-Window Fusion is Crucial: The fusion of features from sampled points across different windows is essential for enabling long-range dependency modeling, a capability lost in purely local window-based approaches.
  • Robustness from Long-Range Context: The paper suggests that the ability to model long-range dependencies inherently contributes to improved model robustness against corruptions and biases, as the model has a broader contextual understanding.

Key Results

  • Classification Accuracy: FaViT-B2 achieved 82.3% Top-1 accuracy on ImageNet-1K, outperforming Swin-T by 1% while using 14% fewer parameters.
  • Efficiency: FaViT exhibits linear computational complexity with respect to image resolution, making it significantly more scalable than ViT and PVTv2 for larger images. For example, at 1000x1000 resolution, FaViT-B2 uses only 25% of the FLOPs of DeiT-S and 50% of PVTv2-B2.
  • Robustness: FaViT shows superior robustness against image corruptions (e.g., FaViT-B2 outperforms Swin-T by 6.6% in accuracy retention on ImageNet-C) and label noise.
  • Downstream Tasks: FaViT variants achieved state-of-the-art results in object detection and semantic segmentation on COCO 2017 and ADE20K, respectively, often outperforming Swin-T with fewer parameters.
  • Ablation Studies: Replacing Swin-T's self-attention with FaSA consistently improved performance and reduced parameters/FLOPs, demonstrating FaSA's effectiveness.

Limitations and Future Work

  • Patch Embedding and Feedforward Layers: The authors acknowledge that FaSA is only applied to the core attention mechanism. They suggest that optimizing the patch embedding and feedforward layers to better match FaSA could further improve performance.
  • Convolution/Pooling Integration: While FaSA focuses on self-attention, they note that other methods explore replacing self-attention with convolution, pooling, or MLPs. Future work could investigate integrating these approaches with FaViT.
  • High-Resolution Image Processing: The paper plans to evaluate FaViT on high-resolution image processing tasks to further demonstrate its value in practical applications.

Summary updated

Abstract

Transformers have astounding representational power but typically consume considerable computation which is quadratic with image resolution. The prevailing Swin transformer reduces computational costs through a local window strategy. However, this strategy inevitably causes two drawbacks: (1) the local window-based self-attention hinders global dependency modeling capability; (2) recent studies point out that local windows impair robustness. To overcome these challenges, we pursue a preferable trade-off between computational cost and performance. Accordingly, we propose a novel factorization self-attention mechanism (FaSA) that enjoys both the advantages of local window cost and long-range dependency modeling capability. By factorizing the conventional attention matrix into sparse sub-attention matrices, FaSA captures long-range dependencies while aggregating mixed-grained information at a computational cost equivalent to the local window-based self-attention. Leveraging FaSA, we present the factorization vision transformer (FaViT) with a hierarchical structure. FaViT achieves high performance and robustness, with linear computational complexity concerning input image spatial resolution. Extensive experiments have shown FaViT's advanced performance in classification and downstream tasks. Furthermore, it also exhibits strong model robustness to corrupted and biased data and hence demonstrates benefits in favor of practical applications. In comparison to the baseline model Swin-T, our FaViT-B2 significantly improves classification accuracy by 1% and robustness by 7%, while reducing model parameters by 14%. Our code will soon be publicly available at https://github.com/q2479036243/FaViT.
BibTeX
@article{qin2023factorization,
  author = {Haolin Qin and Daquan Zhou and Tingfa Xu and Ziyang Bian and Jianan Li},
  title = {{Factorization Vision Transformer: Modeling Long-Range Dependency With Local Window Cost}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2023},
}
Figure