Abstract
Vision Transformers (ViTs) have traditionally been conceptualized as a sequential cascade of Multi-Head Self-Attention (MHSA) and Feed-Forward Network (FFN) layers. This paper proposes a paradigm shift, reimagining the ViT architecture as an ensemble of multiple parallel paths of varying lengths. By unrolling the residual connections inherent in ViTs, the authors demonstrate that the final output is a linear combination of features from these diverse paths. This perspective allows for the identification of "short paths" that contribute noise rather than signal to the final classification. To address this, the researchers introduce EnsembleScale, a method for learnable path re-weighting, and path pruning techniques. These innovations improve the performance, convergence stability, and computational efficiency of both standard and deep ViT architectures, as evidenced by improvements on ImageNet-1K and CIFAR benchmarks.
Introduction
While Vision Transformers have set new benchmarks in computer vision, their internal mechanics often remain a "black box." A significant challenge in scaling these models is performance saturation; as ViTs grow deeper, they frequently encounter optimization difficulties. This paper addresses these issues by questioning the fundamental sequential assumption of ViTs. The authors argue that treating every layer’s output as equally necessary for the final prediction is counterproductive, as shallow representations (short paths) can interfere with the more refined features generated by deeper layers.
The Path Ensemble Perspective
Mathematical Unrolling
The core contribution of this work is the mathematical "unrolling" of the ViT architecture. By analyzing the residual connections, the authors show that a standard serial MHSA $\rightarrow$ FFN block can be rearranged into an equivalent three-path parallel form. When this logic is applied recursively across the entire network, the ViT is revealed to be a sum of outputs from paths of different lengths.
Dual Functionality of Paths
Under this view, every path within a ViT serves two distinct roles:
- Direct Contribution: Providing features directly to the final classifier.
- Representation Support: Providing lower-level representations that serve as the foundation for subsequent, longer paths.
Short Paths as Noise
Through a "toy experiment," the researchers discovered that features from very short paths are nearly orthogonal to the final ensemble feature. In practical terms, this means shallow layers often act as "noise" that the deeper parts of the network must actively suppress to achieve accurate classification. This noise is a primary reason why deep ViTs often fail to converge; the model attempts to "explode" the scale of deep features to drown out the shallow interference.
Methods for Optimization
To leverage the path ensemble view, the authors propose several techniques to refine how features are integrated:
- EnsembleScale: This method introduces a learnable diagonal matrix (per-channel weights) for each path. This allows the model to adaptively determine the optimal contribution of each path to the final result, facilitating better convergence in deep models (e.g., a 24-layer DeiT-S).
- Path Pruning: By identifying and "cutting" underperforming short paths from the final ensemble sum, the authors force shallow layers to focus exclusively on providing features for subsequent layers rather than attempting to influence the final prediction directly.
- Self-Distillation: To optimize the "representation support" role of shallow layers, deeper paths act as "teachers" during training, supervising the shallower paths to ensure they provide high-quality input for the rest of the network.
- High-Pass Filtering: While self-attention typically acts as a low-pass filter (smoothing signals), the proposed path combination strategies act as high-pass filters, helping the model retain critical high-frequency details.
Experimental Results
The proposed methods were validated on ImageNet-1K, CIFAR-10, and CIFAR-100.
- Accuracy Improvements: Integrating EnsembleScale into a standard DeiT-S increased ImageNet accuracy from 79.8% to 80.3% with negligible computational overhead.
- Scaling Depth: An 18-layer DeiT-S equipped with EnsembleScale achieved 81.2% accuracy, significantly outperforming the 12-layer baseline and demonstrating that the ensemble view effectively mitigates the optimization issues of deeper models.
- Dynamic Efficiency: By utilizing the ensemble view to implement "early exit" points, the authors developed a Dynamic ViT. This version reduces computational costs (FLOPs) by 25% while maintaining the original model's accuracy.
- Generalization: Models trained with the ensemble perspective showed superior performance during fine-tuning on CIFAR datasets, suggesting that the path-based optimization leads to more robust and generalized feature learning.
Conclusion and Future Work
Revisiting the Vision Transformer as a path ensemble provides a clearer understanding of the internal dynamics of these models and offers a practical path toward more efficient and deeper architectures. While the current work focuses on vision tasks, the authors suggest that exploring this ensemble view in Natural Language Processing (NLP) is a logical next step. Additionally, future research may involve designing specialized architectures for "short" versus "long" paths now that their distinct roles in the network have been clarified. One remaining challenge is the implementation complexity in hierarchical transformers (like Swin), which requires specific summation strategies to maintain computational efficiency.
Summary updated