Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP

Zhicai Wang, Yanbin Hao, Xingyu Gao, Hao Zhang, Shuo Wang, Tingting Mu, Xiangnan He

TL;DR

Vision MLPs, while promising, struggle to efficiently capture local and global relationships between image tokens, leading to parameter bloat. This paper introduces PosMLP, which replaces the computationally expensive token-mixing layers in vision MLPs with a novel "Positional Spacial Gating Unit" (PoSGU) that leverages relative positional encoding (RPE) to efficiently model cross-token relations, achieving better performance with fewer parameters.

What Problem Does This Paper Address?

Vision Multi-Layer Perceptrons (MLPs) have emerged as a strong alternative to Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) for computer vision tasks. Unlike CNNs that inherently capture local patterns, and ViTs that use self-attention for global interactions, vision MLPs rely on "token-mixing" layers to learn relationships between different parts (tokens) of an image. However, these token-mixing layers, particularly in their standard implementations like gMLP, often have a quadratic complexity with respect to the number of tokens ($O(N^2)$). This leads to a significant increase in the number of parameters, making them less efficient and requiring more data for training. Furthermore, these layers can struggle to effectively capture both fine-grained local details and broader, multi-granular non-local relationships simultaneously, limiting their discriminative power. The paper aims to address this by developing a more parameter-efficient and expressive way to model cross-token relations within vision MLPs.

Methods

The core innovation of this paper is the Positional Spacial Gating Unit (PoSGU), which replaces the standard token-mixing layers in vision MLPs. PoSGU draws inspiration from the Relative Positional Encoding (RPE) techniques used in Vision Transformers.

Here's a breakdown of the key methodological aspects:

  • PoSGU Core Idea: Instead of directly learning pairwise interactions between all tokens (which leads to $O(N^2)$ complexity), PoSGU parameterizes these cross-token relations using RPE. This allows for a more efficient encoding of positional information.
  • RPE Mechanisms: The paper implements two RPE mechanisms within PoSGU:
  • LRPE-based PoSGU: This approach adapts learnable relative positional encoding, reducing the parameter complexity from $O(N^2)$ to $O(N)$.
  • GQPE-based PoSGU: This leverages generalized quadratic positional encoding, achieving a remarkable $O(1)$ parameter complexity, independent of the number of tokens.
  • Group-wise Extension: To further enhance the expressive power and capture multi-granular contexts, the authors propose a group-wise extension. This involves splitting token embeddings into groups and applying the RPE mechanism to each group, allowing for more flexible attention to different granularities of information. This strategy can be applied to both LRPE and GQPE, leading to GLRPE and GGQPE, respectively.
  • PosMLP Architecture: The proposed PoSGU is integrated into a new vision MLP architecture called PosMLP. This architecture also incorporates a convolutional patch embedding (ConvPE) and patch merging (ConvPM) for efficient downsampling and local feature extraction, similar to other modern vision architectures.

Datasets

The paper evaluates its proposed methods on the following datasets:

  • ImageNet1K: Used for image classification tasks. Experiments were conducted with different sample ratios (e.g., 0.5 sample ratio) and resolutions (e.g., 224x224, 384x384).
  • COCO2017: Used for object detection tasks.

Key Insights and Ideas

  • RPE is Key for MLP Efficiency: The paper demonstrates that the principles of RPE, successful in Transformers, can be effectively adapted to vision MLPs to drastically reduce parameter complexity while maintaining or improving performance. This challenges the notion that MLPs inherently require dense, high-parameter token-mixing layers.
  • $O(1)$ Complexity is Achievable: The GGQPE variant of PoSGU achieves $O(1)$ parameter complexity for cross-token relation modeling, which is a significant achievement for efficient vision models.
  • Group-wise RPE Enhances Granularity: The group-wise extension allows the model to capture richer, multi-granular spatial relationships, leading to improved performance. This suggests that simply encoding relative positions might not be enough; learning to attend to different scales of relationships is crucial.
  • Bias Term Mimics Absolute Positional Encoding: The study on the bias term in GGQPE suggests that it can implicitly capture absolute positional information, potentially negating the need for explicit absolute positional encodings in certain contexts, especially when combined with convolutional downsampling.
  • Convolutional Downsampling Complements RPE: The use of convolutional patch embedding and merging modules in PosMLP, which inherently possess translation invariance, works well with the RPE-based token mixing, leading to strong performance.

Key Results

  • ImageNet1K Classification:
    • PosMLP achieved a performance improvement from 72.14% to 74.02% with a parameter reduction from 19.4M to 18.2M on ImageNet1K (0.5 sample ratio) using the GGQPE variant.
  • PosMLP-T achieved 82.1% top-1 accuracy on ImageNet1K (224x224), outperforming CNNs and Transformer-based models like Nest-T and Swin-T in the "tiny models" category.
  • PosMLP-S achieved 83.0% top-1 accuracy, outperforming gMLP-S by a significant margin.
  • PosMLP-B achieved 83.6% top-1 accuracy.
  • Object Detection (COCO2017):
    • PosMLP-T achieved competitive results with state-of-the-art models like PVT-Small and CycleMLP, often with fewer parameters. For instance, PosMLP-T achieved 41.0 AP with 40.5M parameters, compared to CycleMLP-B2's 41.0 AP with 46.5M parameters.
  • PosMLP-S also showed strong performance, achieving 42.1 AP with 47.3M parameters.
  • Parameter Efficiency: The proposed PoSGU significantly reduces parameter complexity, with GGQPE achieving $O(1)$ complexity for cross-token relation modeling.

Limitations and Future Work

  • Pre-training Dataset: The paper notes that for even larger scale pre-training, a more flexible combination between RPE and TokenFC might be necessary, exploring the trade-off between inductive bias and capability.
  • Theoretical Study: The authors hope their work will inspire further theoretical study of positional encoding in vision MLPs.
  • Mature Application: They suggest that their approach could have a mature application similar to how positional encoding is used in Vision Transformers.
  • Bias Term and APE: While the bias term in GGQPE seems to capture absolute positional information, the paper acknowledges that explicit APE might still be beneficial in certain scenarios, particularly for tasks like object detection where translation invariance is critical. The interaction between convolutional downsampling and APE warrants further investigation.

Summary updated

Abstract

Vision multi-layer perceptrons (MLPs) have shown promising performance in computer vision tasks, and become the main competitor of CNNs and vision Transformers. They use token-mixing layers to capture cross-token interactions, as opposed to the multi-head self-attention mechanism used by Transformers. However, the heavily parameterized token-mixing layers naturally lack mechanisms to capture local information and multi-granular non-local relations, thus their discriminative power is restrained. To tackle this issue, we propose a new positional spacial gating unit (PoSGU). It exploits the attention formulations used in the classical relative positional encoding (RPE), to efficiently encode the cross-token relations for token mixing. It can successfully reduce the current quadratic parameter complexity $O(N^2)$ of vision MLPs to $O(N)$ and $O(1)$. We experiment with two RPE mechanisms, and further propose a group-wise extension to improve their expressive power with the accomplishment of multi-granular contexts. These then serve as the key building blocks of a new type of vision MLP, referred to as PosMLP. We evaluate the effectiveness of the proposed approach by conducting thorough experiments, demonstrating an improved or comparable performance with reduced parameter complexity. For instance, for a model trained on ImageNet1K, we achieve a performance improvement from 72.14\% to 74.02\% and a learnable parameter reduction from $19.4M$ to $18.2M$. Code could be found at https://github.com/Zhicaiwww/PosMLP.
BibTeX
@inproceedings{wang2022parameterization,
  author = {Zhicai Wang and Yanbin Hao and Xingyu Gao and Hao Zhang and Shuo Wang and Tingting Mu and Xiangnan He},
  title = {{Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP}},
  booktitle = {ACM Multimedia},
  year = {2022},
  doi = {10.1145/3503161.3547953},
}
Figure