MetaSeg: Content-Aware Meta-Net for Omni-Supervised Semantic Segmentation

Shenwang Jiang, Jianan Li, Ying Wang, Wenxuan Wu, Jizhou Zhang, Bo Huang, Tingfa Xu

Abstract: Noisy labels, inevitably existing in pseudo segmentation labels generated from weak object-level annotations, severely hampers model optimization for semantic segmentation. Previous works often rely on massive hand-crafted losses and carefully-tuned hyper-parameters to resist noise, suffering poor generalization capability and high model complexity. Inspired by recent advances in meta learning, we argue that rather than struggling to tolerate noise hidden behind clean labels passively, a more feasible solution would be to find out the noisy regions actively, so as to simply ignore them during model optimization. With this in mind, this work presents a novel meta learning based semantic segmentation method, MetaSeg, that comprises a primary content-aware meta-net (CAM-Net) to sever as a noise indicator for an arbitrary segmentation model counterpart. Specifically, CAM-Net learns to generate pixel-wise weights to suppress noisy regions with incorrect pseudo labels while highlighting clean ones by exploiting hybrid strengthened features from image content, providing straightforward and reliable guidance for optimizing the segmentation model. Moreover, to break the barrier of time-consuming training when applying meta learning to common large segmentation models, we further present a new decoupled training strategy that optimizes different model layers in a divide-and-conquer manner. Extensive experiments on object, medical, remote sensing and human segmentation shows that our method achieves superior performance, approaching that of fully supervised settings, which paves a new promising way for omni-supervised semantic segmentation.

The Challenge of Noisy Labels

Training high-performance semantic segmentation models typically requires massive amounts of pixel-level annotations—a process that is incredibly labor-intensive. To cut costs, researchers often use "omni-supervised" learning, which mixes a small amount of pixel-perfect data with a large amount of "weakly" labeled data, such as bounding boxes.

However, bounding boxes converted into "pseudo-labels" (using algorithms like GrabCut) are notoriously noisy. For instance, a bounding box for a "cat" will inevitably include background pixels incorrectly labeled as "cat." Standard models struggle because they try to "memorize" these errors. Previous solutions relied on hand-crafted loss functions and tedious hyper-parameter tuning, which often fail to generalize across different domains or noise patterns.

The MetaSeg Framework

MetaSeg addresses this by shifting from passive noise tolerance to active noise identification. The framework consists of two main components:

  1. CAM-Net (Content-Aware Meta-Net): This serves as a "noise indicator." Instead of treating all pseudo-labels as ground truth, CAM-Net learns to generate a pixel-wise weight map. It assigns low weights to "noisy" regions, effectively telling the segmentation model to ignore them during training, while highlighting "clean" regions.
  2. Decoupled Training Strategy: Meta-learning is traditionally slow due to nested optimization loops. MetaSeg introduces a "divide-and-conquer" approach that freezes the bottom layers of the segmentation network during the virtual-train step. This focuses meta-gradients on the top layers, reducing computational overhead by over 6x.
Overview of the MetaSeg training pipeline, showing how CAM-Net generates weight maps to guide the Pixel-Wise Re-weighted Loss.

The "Aha!" Moment: Inconsistency Strengthening

CAM-Net identifies noise by looking for feature inconsistency. High-level CNN features tend to overfit to noisy labels, while low-level features (like edges and textures) remain more faithful to the actual image content. By comparing these multi-level features, CAM-Net can spot where the labels diverge from the visual evidence.

Experimental Results

The authors evaluated MetaSeg across diverse domains, including general objects (PASCAL VOC), urban scenes (Cityscapes), medical imaging (ISIC 2017), and remote sensing.

  • Near-Supervised Performance: On PASCAL VOC 2012, MetaSeg achieved a mean Intersection-over-Union (mIoU) of 69.17%, coming remarkably close to the fully supervised baseline of 70.30%.
  • Efficiency: The decoupled training strategy made the process 6.64x faster than standard meta-learning approaches without sacrificing accuracy.
  • Qualitative Superiority: As shown in the visual results, MetaSeg effectively suppresses noise in the weight maps during training, leading to cleaner segmentation masks at inference compared to baseline methods.
Qualitative comparison showing how MetaSeg generates weight maps to suppress noise during training and produces superior segmentation results at inference.

Conclusion

MetaSeg demonstrates that meta-learning can be both efficient and highly effective for cleaning noisy data in semantic segmentation. By actively identifying noise through feature inconsistency, it provides a robust, domain-agnostic solution that bridges the gap between cheap weak supervision and expensive full supervision.

Summary updated

Abstract

Noisy labels, inevitably existing in pseudo segmentation labels generated from weak object-level annotations, severely hampers model optimization for semantic segmentation. Previous works often rely on massive hand-crafted losses and carefully-tuned hyper-parameters to resist noise, suffering poor generalization capability and high model complexity. Inspired by recent advances in meta learning, we argue that rather than struggling to tolerate noise hidden behind clean labels passively, a more feasible solution would be to find out the noisy regions actively, so as to simply ignore them during model optimization. With this in mind, this work presents a novel meta learning based semantic segmentation method, MetaSeg, that comprises a primary content-aware meta-net (CAM-Net) to sever as a noise indicator for an arbitrary segmentation model counterpart. Specifically, CAM-Net learns to generate pixel-wise weights to suppress noisy regions with incorrect pseudo labels while highlighting clean ones by exploiting hybrid strengthened features from image content, providing straightforward and reliable guidance for optimizing the segmentation model. Moreover, to break the barrier of time-consuming training when applying meta learning to common large segmentation models, we further present a new decoupled training strategy that optimizes different model layers in a divide-and-conquer manner. Extensive experiments on object, medical, remote sensing and human segmentation shows that our method achieves superior performance, approaching that of fully supervised settings, which paves a new promising way for omni-supervised semantic segmentation.
BibTeX
@article{jiang2024metaseg,
  author = {Shenwang Jiang and Jianan Li and Ying Wang and Wenxuan Wu and Jizhou Zhang and Bo Huang and Tingfa Xu},
  title = {{MetaSeg: Content-Aware Meta-Net for Omni-Supervised Semantic Segmentation}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2024},
  url = {https://openreview.net/forum?id=KeYTYkBQ7R},
}
Figure