Weakly Supervised Matrix Factorization for Noisily Tagged Image Parsing

Yulei Niu, Zhiwu Lu, Songfang Huang, Peng Han, Ji-Rong Wen

Abstract

Image parsing, the task of assigning semantic labels to every pixel in an image, typically requires expensive pixel-level supervision. This paper introduces Weakly Supervised Matrix Factorization (WSMF), a framework designed to perform image parsing using only noisy, image-level tags. By formulating the problem as a low-rank matrix factorization with Laplacian regularization, WSMF effectively suppresses noise and achieves state-of-the-art results on standard benchmarks, demonstrating significant robustness to the imperfect tagging common in social media datasets.

Introduction

Image parsing is fundamental to computer vision, requiring both segmentation and classification. Traditional methods rely on thousands of manually annotated pixel-level masks, which are costly to produce. Weakly supervised approaches offer a scalable alternative by using image-level tags. However, real-world tags (e.g., from Flickr) are often "noisy"—containing irrelevant, missing, or incorrect labels. WSMF addresses this challenge by treating image parsing as a noise-reduction problem on a region-label matrix.

Methodology

The WSMF approach follows a structured three-step pipeline:

  1. Over-segmentation: Images are partitioned into small, coherent regions (blobs). Initially, each region is assigned all tags associated with its parent image, resulting in a highly noisy initial label matrix.
  2. Matrix Factorization: The core of the method involves decomposing the noisy label matrix into two low-rank matrices ($U$ and $V$). This low-rank assumption helps filter out incorrect initial assignments by identifying the underlying semantic structure.
  3. Laplacian Regularization: To ensure spatial and visual consistency, a Laplacian regularization term is applied. This constraint forces regions with similar visual features (color, texture) to receive similar labels, even across different images.
The WSMF pipeline: from tagged images to final parsing results via over-segmentation and matrix factorization.

The optimization is performed using an efficient algorithm based on label propagation, which avoids the high computational costs of traditional matrix inversion, making it suitable for large-scale datasets.

Experimental Results

The authors evaluated WSMF on the MSRC and LabelMe datasets. To test robustness, they simulated social media noise by adding up to 100% incorrect tags.

Qualitative parsing results of WSMF compared to ground truth across various image categories.

WSMF achieved a superior accuracy of 71% on the MSRC dataset with clean tags. More impressively, it maintained 55% accuracy even under 100% noise conditions, significantly outperforming competing weakly supervised methods. The model is also computationally efficient, processing the entire MSRC dataset in just 16 seconds on standard hardware. In some instances, WSMF even outperformed fully supervised models in specific categories like "cow" and "sheep."

Conclusion

WSMF provides a robust and efficient solution for image parsing under weak and noisy supervision. By leveraging low-rank matrix structures and visual similarity, it bridges the gap between cheap, noisy tags and high-quality semantic segmentation. While the model's performance is currently capped by the quality of initial over-segmentation and struggles with rare object categories, it represents a significant step toward practical, large-scale image understanding using web-scale data.

Summary updated

Abstract

In this paper, we propose aWeakly Supervised Matrix Factorization (WSMF) approach to the problem of image parsing with noisy tags, i.e., segmenting noisily tagged images and then classifying the regions only with image-level labels. Instead of requiring clean but expensive pixel-level labels as strong supervision in the traditional image parsing methods, we take noisy image-level labels as weakly-supervised constraints. Specifically, we first over-segment all the images into multiple regions which are initially labeled based upon the image-level labels. Moreover, from a low-rank matrix factorization viewpoint, we formulate noisily tagged image parsing as a weakly supervised matrix factorization problem. Finally, we develop an efficient algorithm to solve the matrix factorization problem. Experimental results show the promising performance of the proposed WSMF algorithm in comparison with the state-of-the-arts.
BibTeX
@inproceedings{niu2015weakly,
  author = {Yulei Niu and Zhiwu Lu and Songfang Huang and Peng Han and Ji-Rong Wen},
  title = {{Weakly Supervised Matrix Factorization for Noisily Tagged Image Parsing}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure