Abstract
Image parsing, the task of assigning semantic labels to every pixel in an image, typically requires expensive pixel-level supervision. This paper introduces Weakly Supervised Matrix Factorization (WSMF), a framework designed to perform image parsing using only noisy, image-level tags. By formulating the problem as a low-rank matrix factorization with Laplacian regularization, WSMF effectively suppresses noise and achieves state-of-the-art results on standard benchmarks, demonstrating significant robustness to the imperfect tagging common in social media datasets.
Introduction
Image parsing is fundamental to computer vision, requiring both segmentation and classification. Traditional methods rely on thousands of manually annotated pixel-level masks, which are costly to produce. Weakly supervised approaches offer a scalable alternative by using image-level tags. However, real-world tags (e.g., from Flickr) are often "noisy"—containing irrelevant, missing, or incorrect labels. WSMF addresses this challenge by treating image parsing as a noise-reduction problem on a region-label matrix.
Methodology
The WSMF approach follows a structured three-step pipeline:
- Over-segmentation: Images are partitioned into small, coherent regions (blobs). Initially, each region is assigned all tags associated with its parent image, resulting in a highly noisy initial label matrix.
- Matrix Factorization: The core of the method involves decomposing the noisy label matrix into two low-rank matrices ($U$ and $V$). This low-rank assumption helps filter out incorrect initial assignments by identifying the underlying semantic structure.
- Laplacian Regularization: To ensure spatial and visual consistency, a Laplacian regularization term is applied. This constraint forces regions with similar visual features (color, texture) to receive similar labels, even across different images.
The optimization is performed using an efficient algorithm based on label propagation, which avoids the high computational costs of traditional matrix inversion, making it suitable for large-scale datasets.
Experimental Results
The authors evaluated WSMF on the MSRC and LabelMe datasets. To test robustness, they simulated social media noise by adding up to 100% incorrect tags.
WSMF achieved a superior accuracy of 71% on the MSRC dataset with clean tags. More impressively, it maintained 55% accuracy even under 100% noise conditions, significantly outperforming competing weakly supervised methods. The model is also computationally efficient, processing the entire MSRC dataset in just 16 seconds on standard hardware. In some instances, WSMF even outperformed fully supervised models in specific categories like "cow" and "sheep."
Conclusion
WSMF provides a robust and efficient solution for image parsing under weak and noisy supervision. By leveraging low-rank matrix structures and visual similarity, it bridges the gap between cheap, noisy tags and high-quality semantic segmentation. While the model's performance is currently capped by the quality of initial over-segmentation and struggles with rare object categories, it represents a significant step toward practical, large-scale image understanding using web-scale data.
Summary updated