Introduction
Vision Transformers (ViTs) have established a new state-of-the-art in computer vision tasks, yet their deployment is often hindered by significant computational demands and high parameter counts. While model compression techniques such as pruning and quantization are commonly employed to mitigate these issues, they frequently rely on empirical heuristics that lack a solid theoretical foundation. Furthermore, existing theoretical frameworks like the Neural Tangent Kernel (NTK) are primarily designed for infinite-width models, limiting their applicability to the finite-width architectures used in practice.
This paper introduces the KCR-Transformer, a compact vision transformer block guided by a novel theoretical generalization bound. By minimizing "Kernel Complexity" (KC), the authors propose a principled approach to channel pruning that not only reduces model size but also enhances generalization performance.
Methodology
The KCR-Transformer framework utilizes a two-stage optimization process to achieve efficient and accurate model compression.
Differentiable Channel Selection
During the initial search stage, the model employs Gumbel-Softmax weights to learn the importance of individual channels within the Multi-Layer Perceptron (MLP) layers. This differentiable approach allows the architecture to be optimized directly via standard gradient descent, identifying the most critical features for the task at hand.
Kernel Complexity (KC) Regularization
The core innovation lies in the derivation of a "sharp" generalization bound (Theorem 3.1) that relates the expected loss of a model to its training loss and its Kernel Complexity. To implement this theory in a practical setting, the authors utilize a Nyström-based approximation known as the Truncated Nuclear Norm (TNN). The TNN serves as a regularization term during training, effectively constraining the complexity of the model's internal representations. This ensures that the resulting "KCR-backbones" (e.g., KCR-ViT, KCR-Swin) are both lean and robust.
Experimental Results
The KCR-Transformer was rigorously evaluated on ImageNet-1k for classification, ADE20K for semantic segmentation, and MS-COCO for object detection.
ImageNet-1k Performance
As demonstrated in the comparative analysis, KCR variants consistently outperform original architectures and other state-of-the-art compression methods.
Key highlights from the results include:
- KCR-Swin-B: Achieved a Top-1 accuracy of 84.7%, representing a 1.2% improvement over the baseline Swin-B, while reducing FLOPs by 18% and parameters by approximately 17.8 million.
- KCR-ViT-S: Improved accuracy by 1.0% (82.2% vs 81.2%) while reducing computational load from 4.3G to 3.8G FLOPs.
Downstream Tasks
The benefits of KCR extend beyond classification. On the ADE20K semantic segmentation benchmark, KCR-Swin-B achieved a mean IoU of 52.4, surpassing the standard Swin-B's 51.6. Similar performance gains were observed in object detection and instance segmentation on the MS-COCO dataset, confirming the versatility of the compressed backbones.
Discussion and Conclusion
The KCR-Transformer presents a significant shift from heuristic-based pruning to theory-driven architecture optimization. The "Less is More" paradox observed—where reducing complexity leads to higher accuracy—suggests that principled regularization helps models avoid overfitting and learn more robust representations.
Despite these successes, the method involves a search stage that adds to the initial training time. Additionally, the introduction of hyperparameters such as the balance factor $\eta$ and rank ratio $\gamma$ requires careful tuning. Future research may focus on reducing search overhead and applying KCR to a broader range of architectures, including hybrid models and state-space models. Ultimately, KCR provides a mathematically grounded path toward deploying high-performance vision transformers on resource-constrained devices.
Summary updated