Learning Equivariant Segmentation with Instance-Unique Querying

Wenguan Wang, James Liang, Dongfang Liu

Abstract

Modern instance segmentation has increasingly relied on query-based architectures, where learnable vectors act as dynamic filters to identify and mask objects. However, these models often struggle with discriminative power because they are typically trained to distinguish objects only within the context of a single image. This paper introduces a novel training framework that enhances these models by enforcing two fundamental properties: dataset-level uniqueness and transformation equivariance. By requiring queries to be globally unique across the entire dataset and ensuring internal representations maintain geometric consistency, the proposed method achieves significant performance gains—up to 3.2 AP on the COCO benchmark—without introducing any architectural changes or inference-time overhead.

Introduction

In query-based instance segmentation, models like Mask2Former or SOLOv2 utilize a set of learnable "query vectors." These vectors function as search terms that probe an image's feature maps to localize and segment specific object instances. While effective, current training paradigms suffer from a limitation: queries are only required to be intra-scene discriminative. In a standard training step, a query only needs to distinguish a specific object (e.g., a dog) from other objects in the same image (e.g., a cat or a car).

Because individual training images often contain limited object variety, the queries are never challenged to develop high-level, globally robust representations. This lack of "discriminative muscle" leads to failures in complex, crowded, or ambiguous scenes where objects may share similar visual characteristics. To address this, we propose a two-pronged strategy focusing on inter-scene discrimination and geometric equivariance.

Methodology

The proposed framework is designed as a "plug-and-play" training strategy that can be integrated into existing query-based models. It consists of two primary components used exclusively during the training phase.

Inter-Scene Instance Discrimination

To move beyond local uniqueness, the framework introduces a cross-scene challenge. An external memory bank is employed to store pixel embeddings of object instances from across the entire dataset. During training, a query is tasked not only with identifying its target object in the current image but also with explicitly mismatching all object pixels stored in the memory bank from other images.

To maintain computational efficiency, the system uses sparse, instance-balanced sampling, storing approximately 50 pixels per object. This allows the model to learn a globally unique representation of an instance, effectively sharpening the "search terms" used by the queries to identify objects against a much broader background of possibilities.

Transformation Equivariance

While standard data augmentations like flipping and cropping are common in computer vision, they typically aim for "invariance"—where the output remains the same despite the change. However, in segmentation, spatial location is critical. We argue for equivariance: the principle that if an image undergoes a geometric transformation, the model's internal representations (query embeddings and feature maps) should change in the exact same geometric way.

We implement a regularization loss that enforces this consistency. By ensuring the relationship between a query and its corresponding instance is robust to spatial warping, the model produces more stable and accurate mask predictions.

Crucially, both the memory bank and the equivariance loss are discarded after training. This ensures that the performance improvements come at zero extra computational cost during inference.

Experimental Results

The framework was evaluated on two major benchmarks: MS COCO (80 classes) and LVIS v1 (1,200+ classes), using various backbones including ResNet and Swin Transformer.

Performance Gains

The method demonstrated consistent improvements across multiple state-of-the-art models:

  • SOLOv2: Achieved a jump from 35.1 to 38.3 AP (+3.2 AP) on COCO.
  • Mask2Former: When paired with a Swin-L backbone, the framework set a new record of 51.8 AP on COCO instance segmentation.
  • Generalization: Across four different base models (CondInst, SOLOv2, SOTR, and Mask2Former), the method provided a boost ranging from +1.6 to +3.2 AP.

Long-Tail Robustness

On the LVIS dataset, which tests a model's ability to handle rare object categories, the method improved SOLOv2 by 2.7 AP. Notably, the gains were most pronounced in the "rare" category (+4.0 AP). This suggests that better query discrimination is particularly beneficial when training data for specific classes is scarce, as it prevents the model from misidentifying rare objects as more common ones.

Discussion and Limitations

While the framework significantly advances the state of the art, certain challenges remain. The current equivariance logic is restricted to linear geometric transformations (flipping, rotating, cropping). It does not yet account for photometric changes, such as color jittering or blurring, which are also vital components of robust visual perception. Additionally, the model still faces difficulties with "transparent" objects (e.g., glassware) and extreme occlusions where multiple instances of the same class overlap heavily.

Conclusion

By shifting the focus from local, intra-scene discrimination to global, dataset-level uniqueness and geometric equivariance, we have shown that query-based models possess significant untapped potential. This training philosophy provides a substantial performance boost across various architectures without increasing deployment complexity. Future work will explore extending these principles to other dense prediction tasks, such as panoptic segmentation and object detection.

Summary updated

BibTeX
@inproceedings{wang2022learning,
  author = {Wenguan Wang and James Liang and Dongfang Liu},
  title = {{Learning Equivariant Segmentation with Instance-Unique Querying}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=e824zXt3G1},
}
Figure