Revisiting Vision Transformer from the View of Path Ensemble

Shuning Chang, Pichao Wang, Hao Luo, Fan Wang, Mike Zheng Shou

Abstract

Vision Transformers (ViTs) have traditionally been conceptualized as a sequential cascade of Multi-Head Self-Attention (MHSA) and Feed-Forward Network (FFN) layers. This paper proposes a paradigm shift, reimagining the ViT architecture as an ensemble of multiple parallel paths of varying lengths. By unrolling the residual connections inherent in ViTs, the authors demonstrate that the final output is a linear combination of features from these diverse paths. This perspective allows for the identification of "short paths" that contribute noise rather than signal to the final classification. To address this, the researchers introduce EnsembleScale, a method for learnable path re-weighting, and path pruning techniques. These innovations improve the performance, convergence stability, and computational efficiency of both standard and deep ViT architectures, as evidenced by improvements on ImageNet-1K and CIFAR benchmarks.

Introduction

While Vision Transformers have set new benchmarks in computer vision, their internal mechanics often remain a "black box." A significant challenge in scaling these models is performance saturation; as ViTs grow deeper, they frequently encounter optimization difficulties. This paper addresses these issues by questioning the fundamental sequential assumption of ViTs. The authors argue that treating every layer’s output as equally necessary for the final prediction is counterproductive, as shallow representations (short paths) can interfere with the more refined features generated by deeper layers.

The Path Ensemble Perspective

Mathematical Unrolling

The core contribution of this work is the mathematical "unrolling" of the ViT architecture. By analyzing the residual connections, the authors show that a standard serial MHSA $\rightarrow$ FFN block can be rearranged into an equivalent three-path parallel form. When this logic is applied recursively across the entire network, the ViT is revealed to be a sum of outputs from paths of different lengths.

Dual Functionality of Paths

Under this view, every path within a ViT serves two distinct roles:

  1. Direct Contribution: Providing features directly to the final classifier.
  2. Representation Support: Providing lower-level representations that serve as the foundation for subsequent, longer paths.

Short Paths as Noise

Through a "toy experiment," the researchers discovered that features from very short paths are nearly orthogonal to the final ensemble feature. In practical terms, this means shallow layers often act as "noise" that the deeper parts of the network must actively suppress to achieve accurate classification. This noise is a primary reason why deep ViTs often fail to converge; the model attempts to "explode" the scale of deep features to drown out the shallow interference.

Methods for Optimization

To leverage the path ensemble view, the authors propose several techniques to refine how features are integrated:

  1. EnsembleScale: This method introduces a learnable diagonal matrix (per-channel weights) for each path. This allows the model to adaptively determine the optimal contribution of each path to the final result, facilitating better convergence in deep models (e.g., a 24-layer DeiT-S).
  2. Path Pruning: By identifying and "cutting" underperforming short paths from the final ensemble sum, the authors force shallow layers to focus exclusively on providing features for subsequent layers rather than attempting to influence the final prediction directly.
  3. Self-Distillation: To optimize the "representation support" role of shallow layers, deeper paths act as "teachers" during training, supervising the shallower paths to ensure they provide high-quality input for the rest of the network.
  4. High-Pass Filtering: While self-attention typically acts as a low-pass filter (smoothing signals), the proposed path combination strategies act as high-pass filters, helping the model retain critical high-frequency details.

Experimental Results

The proposed methods were validated on ImageNet-1K, CIFAR-10, and CIFAR-100.

  • Accuracy Improvements: Integrating EnsembleScale into a standard DeiT-S increased ImageNet accuracy from 79.8% to 80.3% with negligible computational overhead.
  • Scaling Depth: An 18-layer DeiT-S equipped with EnsembleScale achieved 81.2% accuracy, significantly outperforming the 12-layer baseline and demonstrating that the ensemble view effectively mitigates the optimization issues of deeper models.
  • Dynamic Efficiency: By utilizing the ensemble view to implement "early exit" points, the authors developed a Dynamic ViT. This version reduces computational costs (FLOPs) by 25% while maintaining the original model's accuracy.
  • Generalization: Models trained with the ensemble perspective showed superior performance during fine-tuning on CIFAR datasets, suggesting that the path-based optimization leads to more robust and generalized feature learning.

Conclusion and Future Work

Revisiting the Vision Transformer as a path ensemble provides a clearer understanding of the internal dynamics of these models and offers a practical path toward more efficient and deeper architectures. While the current work focuses on vision tasks, the authors suggest that exploring this ensemble view in Natural Language Processing (NLP) is a logical next step. Additionally, future research may involve designing specialized architectures for "short" versus "long" paths now that their distinct roles in the network have been clarified. One remaining challenge is the implementation complexity in hierarchical transformers (like Swin), which requires specific summation strategies to maintain computational efficiency.

Summary updated

Abstract

Vision Transformers (ViTs) are normally regarded as a stack of transformer layers. In this work, we propose a novel view of ViTs showing that they can be seen as ensemble networks containing multiple parallel paths with different lengths. Specifically, we equivalently transform the traditional cascade of multi-head self-attention (MSA) and feed-forward network (FFN) into three parallel paths in each transformer layer. Then, we utilize the identity connection in our new transformer form and further transform the ViT into an explicit multi-path ensemble network. From the new perspective, these paths perform two functions: the first is to provide the feature for the classifier directly, and the second is to provide the lower-level feature representation for subsequent longer paths. We investigate the influence of each path for the final prediction and discover that some paths even pull down the performance. Therefore, we propose the path pruning and EnsembleScale skills for improvement, which cut out the underperforming paths and re-weight the ensemble components, respectively, to optimize the path combination and make the short paths focus on providing high-quality representation for subsequent paths. We also demonstrate that our path combination strategies can help ViTs go deeper and act as high-pass filters to filter out partial low-frequency signals. To further enhance the representation of paths served for subsequent paths, self-distillation is applied to transfer knowledge from the long paths to the short paths. This work calls for more future research to explain and design ViTs from new perspectives.
BibTeX
@inproceedings{chang2023revisiting,
  author = {Shuning Chang and Pichao Wang and Hao Luo and Fan Wang and Mike Zheng Shou},
  title = {{Revisiting Vision Transformer from the View of Path Ensemble}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure