MP-Former: Mask-Piloted Transformer for Image Segmentation

Hao Zhang, Feng Li, Huaizhe Xu, Shijia Huang, Shilong Liu, Lionel M. Ni, Lei Zhang

Introduction

In recent years, Transformer-based architectures have redefined the state-of-the-art in image segmentation. Among these, Mask2Former stands out by utilizing a "masked attention" mechanism. This approach uses predicted masks from one decoder layer to provide spatial constraints for the subsequent layer, theoretically allowing the model to focus on specific object regions. However, despite its success, Mask2Former suffers from significant training instabilities.

This paper introduces MP-Former, a novel framework that addresses these instabilities through a "mask-piloted" training strategy. By stabilizing the optimization goals across decoder layers, MP-Former achieves faster convergence and superior accuracy without adding any computational overhead during inference.

The Problem: Layer-wise Inconsistency

The authors identify a critical flaw in the standard Mask2Former training process: layer-wise inconsistency. In Mask2Former, each decoder layer predicts masks independently via a dot-product between queries and image features. Because these predictions can change drastically from one layer to the next—a phenomenon referred to as "jitter"—the model faces inconsistent optimization targets during the early stages of training.

To quantify this, the authors introduced two new metrics:

  1. mIoU-L (Layer-wise mean IoU): Measures the consistency of mask predictions across consecutive decoder layers.
  2. Util (Query Utilization): Tracks how many queries are effectively assigned to ground-truth objects.

Their analysis revealed that high jitter leads to low query utilization, forcing the model into a "cold start" period where it struggles to find objects, resulting in slow convergence and sub-optimal final performance.

Methodology: Mask-Piloted Training

MP-Former introduces Mask-Piloted (MP) training, a training-only modification designed to anchor the learning process. The core idea is to provide the decoder with "piloted" queries that have a clear, stable relationship with the ground truth.

1. Query Splitting and Piloting

During training, the decoder queries are divided into two distinct groups. The first group follows the standard Mask2Former matching process. The second group—the MP group—is initialized with ground-truth (GT) class embeddings and guided by GT masks used as attention masks. This setup transforms the segmentation task into a reconstruction or denoising task for the MP group.

2. Noised Masks for Robustness

To ensure the model does not simply memorize the ground truth, the authors introduce noise into the pilot masks. Specifically, they utilize point noise, which involves randomly flipping pixels within the mask. This forces the model to learn how to refine and denoise inaccurate spatial information—a skill that directly translates to better performance when processing the predicted (and naturally noisy) masks of the standard query group.

3. Zero-Cost Inference

A key advantage of MP-Former is its efficiency. The MP query group and the associated reconstruction loss are only utilized during the training phase. For deployment, the MP components are discarded, leaving a model with the exact same architecture, parameter count, and FLOPs as the original Mask2Former.

Experimental Results

MP-Former was evaluated on major benchmarks including ADE20K, Cityscapes, and MS COCO 2017. The results demonstrate that stabilizing the training process yields substantial dividends:

  • Accuracy Improvements: On the Cityscapes dataset, MP-Former achieved a +2.3 AP increase in instance segmentation and a +1.6 mIoU improvement in semantic segmentation using a ResNet-50 backbone.
  • Accelerated Convergence: On ADE20K, the model reached performance levels superior to the baseline in just 80k steps, effectively halving the required training time (160k steps) of the original Mask2Former.
  • Scalability: The benefits of MP training were consistent across various backbones, from standard ResNet-50 to high-capacity models like Swin-Large.

Discussion and Insights

The research highlights that the type of noise used in training is critical. While previous object detection models (like DN-DETR) benefited from box-shifting noise, MP-Former found that pixel-level point noise is significantly more effective for the fine-grained requirements of mask segmentation. This suggests that the relationship between spatial coordinates and mask features in Transformers is highly sensitive to local pixel distribution.

Conclusion

MP-Former demonstrates that addressing layer-wise inconsistency is a powerful lever for improving Transformer-based segmentation. By introducing a mask-piloted denoising task during training, the model achieves a more stable optimization path and higher query utilization. The result is a state-of-the-art segmentation tool that is faster to train and more accurate to deploy, all while maintaining the inference efficiency of its predecessors.

Summary updated

Abstract

We present a mask-piloted Transformer which improves masked-attention in Mask2Former for image segmentation. The improvement is based on our observation that Mask2Former suffers from inconsistent mask predictions between consecutive decoder layers, which leads to inconsistent optimization goals and low utilization of decoder queries. To address this problem, we propose a mask-piloted training approach, which additionally feeds noised ground-truth masks in masked-attention and trains the model to reconstruct the original ones. Compared with the predicted masks used in mask-attention, the ground-truth masks serve as a pilot and effectively alleviate the negative impact of inaccurate mask predictions in Mask2Former. Based on this technique, our \M achieves a remarkable performance improvement on all three image segmentation tasks (instance, panoptic, and semantic), yielding $+2.3$AP and $+1.6$mIoU on the Cityscapes instance and semantic segmentation tasks with a ResNet-50 backbone. Our method also significantly speeds up the training, outperforming Mask2Former with half of the number of training epochs on ADE20K with both a ResNet-50 and a Swin-L backbones. Moreover, our method only introduces little computation during training and no extra computation during inference. Our code will be released at \url{https://github.com/IDEA-Research/MP-Former}.
BibTeX
@inproceedings{zhang2023mp,
  author = {Hao Zhang and Feng Li and Huaizhe Xu and Shijia Huang and Shilong Liu and Lionel M. Ni and Lei Zhang},
  title = {{MP-Former: Mask-Piloted Transformer for Image Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure