Dynamically pruning segformer for efficient semantic segmentation

Haoli Bai, Hongda Mao, Dinesh Nair

Abstract

As one of the successful Transformer-based models in computer vision tasks, SegFormer demonstrates superior performance in semantic segmentation. Nevertheless, the high computational cost greatly challenges the deployment of SegFormer on edge devices. In this paper, we seek to design a lightweight SegFormer for efficient semantic segmentation. Based on the observation that neurons in SegFormer layers exhibit large variances across different images, we propose a dynamic gated linear layer, which prunes the most uninformative set of neurons based on the input instance. To improve the dynamically pruned SegFormer, we also introduce two-stage knowledge distillation to transfer the knowledge within the original teacher to the pruned student network. Experimental results show that our method can significantly reduce the computation overhead of SegFormer without an apparent performance drop. For instance, we can achieve 36.9% mIoU with only 3.3G FLOPs on ADE20K, saving more than 60% computation with the drop of only 0.5% in mIoU.

Introduction

Semantic segmentation is a critical yet computationally intensive task, requiring pixel-level classification. While Vision Transformer (ViT) architectures like SegFormer have achieved state-of-the-art results, their reliance on dense, wide fully connected layers makes them difficult to deploy on resource-constrained edge devices. These models typically apply the same computational load to every image, regardless of complexity. This research identifies a significant inefficiency: many neurons within SegFormer's encoder are redundant or only relevant to specific input types. By dynamically pruning these uninformative neurons, we can drastically reduce FLOPs without sacrificing accuracy.

The Dynamic Gated Linear Layer (DGL)

The core innovation of DynaSegFormer is the Dynamic Gated Linear Layer (DGL). Unlike static pruning, which removes the same neurons for every input, DGL utilizes a lightweight Gate Predictor ($g_\phi(\cdot)$). This predictor analyzes the input features to determine which neurons are most informative for the current image.

The architecture of the Dynamic Gated Linear Layer and its Gate Predictor module.

As shown in the diagram, the Gate Predictor employs a Top-K selection mechanism to identify the top $r\%$ of neurons. It generates a binary mask $\mathbf{M}$ that is applied to the weight matrices $\mathbf{W}_1$ and $\mathbf{W}_2$. This ensures that only the most relevant computations are performed during the forward pass, effectively skipping the "useless" math for a given instance.

Neuron Taxonomy and Motivation

The motivation for dynamic pruning stems from an analysis of neuron behavior across diverse datasets. The authors categorized neurons into three distinct types based on their activation magnitudes:

  • Type-I (Always On): Consistently informative across nearly all images.
  • Type-II (Input-Dependent): Importance varies significantly depending on the specific content of the image.
  • Type-III (Always Off): Generally uninformative regardless of the input.
Box plot showing the magnitude distribution of neurons, categorized into Type-I, II, and III.

This taxonomy highlights why dynamic pruning is superior to static methods; static pruning would likely remove Type-II neurons that are critical for specific edge cases, whereas DGL can retain them only when necessary.

Training Strategy

To maintain performance after pruning, the authors implemented a Two-Stage Knowledge Distillation (KD) framework. In the first stage, the student (DynaSegFormer) learns to mimic the intermediate features of the unpruned teacher's encoder. In the second stage, the student is refined to match the teacher's final logit predictions. Additionally, Sparsity Annealing is used to gradually increase the pruning ratio during training, preventing the model from collapsing due to sudden information loss.

Key Results

DynaSegFormer was evaluated on the ADE20K and Cityscapes benchmarks, demonstrating substantial efficiency gains:

  • Efficiency on ADE20K: The model achieved 36.9% mIoU using only 3.3G FLOPs. Compared to the baseline SegFormer (37.4% mIoU at 8.4G FLOPs), this represents a 60% reduction in computation with a negligible 0.5% drop in accuracy.
  • Dynamic vs. Static: Dynamic pruning outperformed static pruning by nearly 3% mIoU on ADE20K, validating the per-image selection strategy.
  • Comparison to CNNs: DynaSegFormer outperformed DeepLabV3+ by 2.9% mIoU while requiring only 5% of the FLOPs.

Conclusion

DynaSegFormer successfully addresses the computational bottleneck of Transformer-based segmentation models. While the dynamic approach slightly increases the total parameter count (by ~12%) to store the gate predictors, the massive reduction in inference-time FLOPs makes it a viable candidate for real-time applications on mobile and edge hardware. Future work may involve extending this dynamic gating mechanism to other Transformer components beyond the linear layers.

Summary updated

Abstract

As one of the successful Transformer-based models in computer vision tasks, SegFormer demonstrates superior performance in semantic segmentation. Nevertheless, the high computational cost greatly challenges the deployment of SegFormer on edge devices. In this paper, we seek to design a lightweight SegFormer for efficient semantic segmentation. Based on the observation that neurons in SegFormer layers exhibit large variances across different images, we propose a dynamic gated linear layer, which prunes the most uninformative set of neurons based on the input instance. To improve the dynamically pruned SegFormer, we also introduce two-stage knowledge distillation to transfer the knowledge within the original teacher to the pruned student network. Experimental results show that our method can significantly reduce the computation overhead of SegFormer without an apparent performance drop. For instance, we can achieve 36.9% mIoU with only 3.3G FLOPs on ADE20K, saving more than 60% computation with the drop of only 0.5% in mIoU
BibTeX
@inproceedings{bai2022dynamically,
  author = {Haoli Bai and Hongda Mao and Dinesh Nair},
  title = {{Dynamically pruning segformer for efficient semantic segmentation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=8rqJExw3DX},
}
Figure