Compact Vision Transformer by Reduction of Kernel Complexity

Yancheng Wang, Yingzhen YangarXiv 2025 · DOI · arXiv

Introduction

Vision Transformers (ViTs) have established a new state-of-the-art in computer vision tasks, yet their deployment is often hindered by significant computational demands and high parameter counts. While model compression techniques such as pruning and quantization are commonly employed to mitigate these issues, they frequently rely on empirical heuristics that lack a solid theoretical foundation. Furthermore, existing theoretical frameworks like the Neural Tangent Kernel (NTK) are primarily designed for infinite-width models, limiting their applicability to the finite-width architectures used in practice.

This paper introduces the KCR-Transformer, a compact vision transformer block guided by a novel theoretical generalization bound. By minimizing "Kernel Complexity" (KC), the authors propose a principled approach to channel pruning that not only reduces model size but also enhances generalization performance.

Methodology

The KCR-Transformer framework utilizes a two-stage optimization process to achieve efficient and accurate model compression.

Differentiable Channel Selection

During the initial search stage, the model employs Gumbel-Softmax weights to learn the importance of individual channels within the Multi-Layer Perceptron (MLP) layers. This differentiable approach allows the architecture to be optimized directly via standard gradient descent, identifying the most critical features for the task at hand.

Kernel Complexity (KC) Regularization

The core innovation lies in the derivation of a "sharp" generalization bound (Theorem 3.1) that relates the expected loss of a model to its training loss and its Kernel Complexity. To implement this theory in a practical setting, the authors utilize a Nyström-based approximation known as the Truncated Nuclear Norm (TNN). The TNN serves as a regularization term during training, effectively constraining the complexity of the model's internal representations. This ensures that the resulting "KCR-backbones" (e.g., KCR-ViT, KCR-Swin) are both lean and robust.

Experimental Results

The KCR-Transformer was rigorously evaluated on ImageNet-1k for classification, ADE20K for semantic segmentation, and MS-COCO for object detection.

ImageNet-1k Performance

As demonstrated in the comparative analysis, KCR variants consistently outperform original architectures and other state-of-the-art compression methods.

Table 1: Performance comparisons on ImageNet-1k. KCR models (Ours) consistently show higher accuracy with lower computational costs.

Key highlights from the results include:

  • KCR-Swin-B: Achieved a Top-1 accuracy of 84.7%, representing a 1.2% improvement over the baseline Swin-B, while reducing FLOPs by 18% and parameters by approximately 17.8 million.
  • KCR-ViT-S: Improved accuracy by 1.0% (82.2% vs 81.2%) while reducing computational load from 4.3G to 3.8G FLOPs.

Downstream Tasks

The benefits of KCR extend beyond classification. On the ADE20K semantic segmentation benchmark, KCR-Swin-B achieved a mean IoU of 52.4, surpassing the standard Swin-B's 51.6. Similar performance gains were observed in object detection and instance segmentation on the MS-COCO dataset, confirming the versatility of the compressed backbones.

Discussion and Conclusion

The KCR-Transformer presents a significant shift from heuristic-based pruning to theory-driven architecture optimization. The "Less is More" paradox observed—where reducing complexity leads to higher accuracy—suggests that principled regularization helps models avoid overfitting and learn more robust representations.

Despite these successes, the method involves a search stage that adds to the initial training time. Additionally, the introduction of hyperparameters such as the balance factor $\eta$ and rank ratio $\gamma$ requires careful tuning. Future research may focus on reducing search overhead and applying KCR to a broader range of architectures, including hybrid models and state-space models. Ultimately, KCR provides a mathematically grounded path toward deploying high-performance vision transformers on resource-constrained devices.

Summary updated

Abstract

Self-attention and transformer architectures have become foundational components in modern deep learning. Recent efforts have integrated transformer blocks into compact neural architectures for computer vision, giving rise to various efficient vision transformers. In this work, we introduce Transformer with Kernel Complexity Reduction, or KCR-Transformer, a compact transformer block equipped with differentiable channel selection, guided by a novel and sharp theoretical generalization bound. KCR-Transformer performs input/output channel selection in the MLP layers of transformer blocks to reduce the computational cost. Furthermore, we provide a rigorous theoretical analysis establishing a tight generalization bound for networks equipped with KCR-Transformer blocks. Leveraging such strong theoretical results, the channel pruning by KCR-Transformer is conducted in a generalization-aware manner, ensuring that the resulting network retains a provably small generalization error. Our KCR-Transformer is compatible with many popular and compact transformer networks, such as ViT and Swin, and it reduces the FLOPs of the vision transformers while maintaining or even improving the prediction accuracy. In the experiments, we replace all the transformer blocks in the vision transformers with KCR-Transformer blocks, leading to KCR-Transformer networks with different backbones. The resulting TCR-Transformers achieve superior performance on various computer vision tasks, achieving even better performance than the original models with even less FLOPs and parameters.
BibTeX
@misc{wang2025compact,
  author = {Yancheng Wang and Yingzhen Yang},
  title = {{Compact Vision Transformer by Reduction of Kernel Complexity}},
  year = {2025},
  eprint = {2507.12780},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.12780},
}
Figure