Dynamically Pruning Segformer for Efficient Semantic Segmentation

Haoli Bai, Hongda Mao, Dinesh Nair

TL;DR

Researchers have developed DynaSegFormer, a version of the popular SegFormer model that uses a "dynamic gated linear layer" to prune uninformative neurons on the fly. By identifying which neurons are actually needed for a specific image, the model can slash computational costs by over 60% with a negligible 0.5% drop in accuracy, making high-quality semantic segmentation feasible for low-power edge devices.

What Problem Does This Paper Address?

Semantic segmentation—the task of assigning a class label (like "car," "tree," or "road") to every single pixel in an image—is computationally expensive. While Vision Transformer (ViT) based models like SegFormer have set new performance benchmarks, they are "heavy." Their encoders rely on wide, dense fully connected layers that perform the same massive amount of math regardless of whether the input image is a simple empty road or a complex, crowded city intersection.

This "one-size-fits-all" approach makes it difficult to deploy these models on edge devices like smartphones or wearable tech. The authors noticed a key inefficiency: many neurons in the SegFormer encoder are redundant or only useful for specific types of images. If we could skip the "useless" math for a given image, we could make the model much faster without losing detail.

Methods

The authors introduce two primary innovations to create DynaSegFormer:

  1. Dynamic Gated Linear Layer (DGL): Instead of a standard static layer, the DGL uses a lightweight "gate predictor." This predictor looks at the input image and decides which neurons are most relevant. It uses a Top-r selection mechanism to keep only the most informative $r\%$ of neurons and masks the rest. This reduces the actual matrix multiplication required during inference.
  2. Two-Stage Knowledge Distillation: Pruning a model usually hurts its performance. To fix this, the authors use a "teacher" (the original, unpruned SegFormer) to train the "student" (the pruned DynaSegFormer).
  3. Stage 1: The student learns to mimic the intermediate features of the teacher's encoder.
  4. Stage 2: The student learns to match the teacher’s final output predictions (logits).
  5. Sparsity Annealing: To prevent the model from "crashing" by pruning too much too early, they gradually increase the pruning intensity during the training process.

Datasets

The model was evaluated on two gold-standard benchmarks for semantic segmentation:

  • ADE20K: A challenging dataset containing 150 semantic classes (indoor and outdoor scenes).
  • Cityscapes: A dataset focused on urban street scenes recorded from a car's perspective.

Key Insights and Ideas

  • Neuron Taxonomy: The authors categorized neurons into three types based on their behavior across thousands of images:
  • Type-I (Always On): Informative across almost all images.
  • Type-II (Input-Dependent): Their importance changes wildly depending on what’s in the picture.
  • Type-III (Always Off): Mostly useless regardless of the input.
  • Dynamic vs. Static: The paper proves that dynamic pruning (deciding what to prune per image) is significantly better than static pruning (removing the same neurons for every image). On ADE20K, dynamic pruning outperformed static pruning by nearly 3% mIoU.
  • The "Aha" Moment: The gate predictor is incredibly cheap to run—its computational cost is negligible compared to the savings it generates in the main linear layers.

Key Results

The results demonstrate a massive efficiency gain:

  • Efficiency: On the ADE20K dataset, DynaSegFormer achieved 36.9% mIoU (mean Intersection over Union) using only 3.3G FLOPs. Compared to the original SegFormer (37.4% mIoU at 8.4G FLOPs), it saved over 60% of the computation with only a 0.5% performance dip.
  • Comparison to CNNs: DynaSegFormer outperformed established convolutional models like DeepLabV3+ by 2.9% mIoU while using only 5% of the FLOPs.
  • Real-Time Capability: By using the MiT-B0 backbone, the model reaches a level of efficiency suitable for real-time applications on mobile hardware.

Limitations and Future Work

  • Parameter Count: While the computation (FLOPs) decreases, the number of parameters actually increases slightly (about 12%) because the model needs to store the weights for the gate predictors and keep all original weights available (since any neuron could be called upon for a specific image). This means the model takes up more storage space even if it runs faster.
  • Hardware Support: Dynamic pruning requires specialized software kernels to see the full speedup on actual hardware, as standard deep learning frameworks are often optimized for static, dense matrices.
  • Future Directions: The authors suggest this dynamic approach could be extended beyond the encoder to other parts of the transformer architecture or applied to different computer vision tasks like object detection.

Summary updated

Abstract

As one of the successful Transformer-based models in computer vision tasks, SegFormer demonstrates superior performance in semantic segmentation. Nevertheless, the high computational cost greatly challenges the deployment of SegFormer on edge devices. In this paper, we seek to design a lightweight SegFormer for efficient semantic segmentation. Based on the observation that neurons in SegFormer layers exhibit large variances across different images, we propose a dynamic gated linear layer, which prunes the most uninformative set of neurons based on the input instance. To improve the dynamically pruned SegFormer, we also introduce two-stage knowledge distillation to transfer the knowledge within the original teacher to the pruned student network. Experimental results show that our method can significantly reduce the computation overhead of SegFormer without an apparent performance drop. For instance, we can achieve 36.9% mIoU with only 3.3G FLOPs on ADE20K, saving more than 60% computation with the drop of only 0.5% in mIoU.
BibTeX
@inproceedings{bai2022dynamically,
  author = {Haoli Bai and Hongda Mao and Dinesh Nair},
  title = {{Dynamically Pruning Segformer for Efficient Semantic Segmentation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=8rqJExw3DX},
}

Loading related content...

Figure