AsyInst: Asymmetric Affinity with DepthGrad and Color for Box-Supervised Instance Segmentation

Siwei Yang, Longlong Jing, Junfei Xiao, Hang Zhao, Alan L. Yuille, Yingwei LiarXiv 2022 · DOI · OpenReview

TL;DR

This paper introduces AsyInst, a novel approach to box-supervised instance segmentation that tackles limitations in existing pairwise affinity losses. By proposing an asymmetric affinity loss, AsyInst effectively prevents trivial, box-shaped predictions and generalizes better across different modalities like depth gradients, leading to state-of-the-art performance on the Cityscapes dataset.

What Problem Does This Paper Address?

Instance segmentation, a task that involves both object detection and pixel-level mask prediction, is crucial for many computer vision applications. However, obtaining the fine-grained pixel-wise annotations required for training is extremely expensive and time-consuming. To mitigate this, researchers have explored "box-supervised" methods that use only bounding boxes as supervision. A common technique in these methods is to employ pairwise affinity losses, which encourage pixels within an object to have similar predicted labels.

The authors identify two key problems with existing pairwise affinity losses, particularly the color affinity loss used in methods like BoxInst:

  1. Limited Modality Support: The symmetric nature of the loss works well with color information but performs poorly when applied to other modalities, such as depth gradients, which are known to be good at identifying object boundaries.
  2. Trivial Predictions: Counter-intuitively, the symmetric affinity loss, intended to prevent trivial predictions (like perfect bounding box masks), actually exacerbates this problem. This is due to an unfavorable interaction with the projection loss, which biases the model towards predicting false positives within the bounding box.

Methods

The core contribution of this paper is the Asymmetric Affinity Loss. The authors propose a simple yet effective modification to the standard symmetric pairwise affinity loss. By introducing an offset parameter ($\delta$) into the loss function, they shift the landscape of the affinity loss. This asymmetry encourages pixel pairs to fall into the "double-negative" region of the loss landscape, effectively penalizing trivial predictions and compensating for the bias introduced by the projection loss.

To further address potential issues like gradient vanishing caused by large asymmetry, they also introduce a smoothness parameter ($\gamma$) that modulates the loss. This asymmetric affinity loss is designed to be general and can be applied to various modalities, including color and depth gradients.

The paper also leverages Depth Gradient Affinity, utilizing the intuition that depth gradients are more consistent near object boundaries. They define a depth gradient affinity measure and a corresponding loss function that is analogous to the color affinity loss.

The overall framework, AsyInst, combines the proposed asymmetric affinity loss (for both color and depth gradients) with the projection loss, which is standard in box-supervised methods.

Datasets

  • Cityscapes: Used for training and evaluation of the proposed AsyInst method. Disparity maps from Cityscapes were converted to depth maps.
  • iSAID: Used for additional experiments to demonstrate the effectiveness of AsyInst on a dataset with small objects and complex backgrounds.

Key Insights and Ideas

  • Symmetry is the Enemy: The paper convincingly demonstrates that the symmetry of existing pairwise affinity losses is the root cause of their failure to prevent trivial predictions and their poor generalization to modalities like depth gradients.
  • Asymmetry as a Solution: Introducing asymmetry into the affinity loss is a simple yet powerful way to counteract the biases of other losses (like projection loss) and guide the model towards more accurate, non-trivial predictions.
  • Depth Gradients are Underrated: The work highlights the potential of using depth gradient information for instance segmentation, showing that it can complement color information and improve performance, especially when paired with an appropriate affinity loss.
  • Generalizable Affinity Loss: The proposed asymmetric affinity loss is presented as a general framework that can be applied to different modalities, making it a versatile tool for box-supervised instance segmentation.

Key Results

  • Cityscapes Performance: AsyInst achieves state-of-the-art results on the Cityscapes dataset, outperforming previous methods. Specifically, it achieves a 3.54% improvement in mask AP compared to the baseline method (BoxInst with symmetric color affinity) when using both color and depth gradient information.
  • Ablation Studies:
    • The asymmetric depth-gradient affinity loss significantly outperforms the symmetric version.
  • Asymmetric color affinity loss also shows improvements over its symmetric counterpart, indicating that symmetry is a general limitation.
  • Qualitative results (Figure 7 and 9) clearly show that AsyInst with asymmetric affinity loss produces more accurate masks and effectively prevents the generation of trivial box-shaped predictions, especially when using depth gradients.
  • iSAID Performance: AsyInst also achieves the highest performance on the iSAID dataset, outperforming strong baselines like BoxLevelSet.

Limitations and Future Work

  • Upper Limit for $\gamma$: The paper notes that a very large $\gamma$ can lead to unwanted local minima and suggests an upper limit for $\gamma$ (approximately 2.71) to ensure stable convergence.
  • Future Exploration: The authors believe their work can inspire further research into pairwise affinity regularization with various modalities, suggesting that their asymmetric approach is a promising direction for future work in weakly supervised instance segmentation.

Summary updated

Abstract

The weakly supervised instance segmentation is a challenging task. The existing methods typically use bounding boxes as supervision and optimize the network with a regularization loss term such as pairwise color affinity loss for instance segmentation. Through systematic analysis, we found that the commonly used pairwise affinity loss has two limitations: (1) it works with color affinity but leads to inferior performance with other modalities such as depth gradient, (2)the original affinity loss does not prevent trivial predictions as intended but actually accelerates this process due to the affinity loss term being symmetric. To overcome these two limitations, in this paper, we propose a novel asymmetric affinity loss which provides the penalty against the trivial prediction and generalizes well with affinity loss from different modalities. With the proposed asymmetric affinity loss, our method outperforms the state-of-the-art methods on the Cityscapes dataset and outperforms our baseline method by 3.5% in mask AP.
BibTeX
@misc{yang2022asyinst,
  author = {Siwei Yang and Longlong Jing and Junfei Xiao and Hang Zhao and Alan L. Yuille and Yingwei Li},
  title = {{AsyInst: Asymmetric Affinity with DepthGrad and Color for Box-Supervised Instance Segmentation}},
  year = {2022},
  eprint = {2212.03517},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.03517},
}
Figure