Social Image Parsing by Cross-Modal Data Refinement

Zhiwu Lu, Xin Gao, Songfang Huang, Liwei Wang, Ji-Rong Wen

Introduction

Image parsing—the task of segmenting and identifying all objects within an image—is a cornerstone of computer vision. Traditionally, this requires massive datasets with pixel-level manual annotations, which are prohibitively expensive and time-consuming to produce. In contrast, "social images" from platforms like Flickr are abundant and come with user-provided tags. However, these tags are both "weak" (lacking spatial information) and "noisy" (containing errors or omissions).

This paper introduces Cross-Modal Data Refinement (CMDR), a novel algorithm designed to bridge this gap. CMDR performs high-quality image parsing using only noisy, image-level social tags, achieving results that rival fully supervised methods while maintaining high computational efficiency.

Methodology

The proposed pipeline transforms weak, noisy image-level tags into precise pixel-level segmentations through a multi-stage process.

Overview of the CMDR pipeline, showing the transition from raw images and tags to final parsing results via oversegmentation and iterative refinement.

1. Oversegmentation and Initial Labeling

The process begins by breaking each image into multiple homogeneous regions using a modified "Blobworld" method. To handle the initial noise of assigning image-level tags to every sub-region, the authors apply two smoothing techniques:

  • Region Size Weighting: Larger regions are prioritized, as smaller segments are more likely to be noise.
  • Semantic Context: A Pearson correlation matrix of tags helps identify likely co-occurrences (e.g., "sky" and "mountain"), providing a prior for label distribution.

2. Cross-Modal Data Refinement (CMDR)

The core of the approach is the CMDR algorithm, which treats visual features and labels as distinct modalities. It formulates parsing as an optimization problem with three key components:

  • Frobenius-norm fitting: Maintains consistency between predicted labels and visual features.
  • Laplacian regularization: Ensures that visually similar regions across the dataset receive consistent labels.
  • $L_1$-norm fitting: Crucially, the $L_1$-norm is used for its robustness to noise, effectively filtering out "garbage" tags that do not correspond to visual patterns.

By iteratively updating the relationship between visual features and labels, the model learns to ignore irrelevant tags and refine the spatial boundaries of objects.

Experimental Evaluation

The researchers evaluated CMDR on three benchmarks: MSRC (591 images, 21 categories), LabelMe/SIFT Flow (2,688 images, 33 categories), and a custom Flickr Dataset derived from PASCAL VOC 2007 using real-world noisy tags. To rigorously test robustness, they manually injected varying levels of random noise—up to 100% additional incorrect labels—into the MSRC and LabelMe datasets.

Key Results

CMDR demonstrated significant improvements over existing weakly supervised methods in both accuracy and noise tolerance.

Quantitative comparison of CMDR against state-of-the-art methods across different noise levels on the MSRC and LabelMe datasets.
  • High Accuracy: On the MSRC dataset with clean tags, CMDR achieved 74% accuracy, outperforming competitors by a wide margin.
  • Robustness to Noise: Even with 100% added noise, CMDR maintained a 59% accuracy on MSRC. In comparison, other methods like [Liu et al., 2013] saw performance drop to 38% under the same conditions.
  • Efficiency: CMDR is remarkably fast, processing the MSRC dataset (7,500 regions) in just 23 seconds—roughly 2x to 3x faster than previous data refinement techniques.
  • Closing the Supervision Gap: The performance of CMDR under noisy, weak supervision (59-74%) is remarkably close to fully supervised methods (64-76%), which require far more expensive training data.

Conclusion and Future Work

CMDR provides a powerful framework for leveraging the vast, "free" data available on social media for complex vision tasks. By treating labels and features as mutually refining modalities and employing robust $L_1$ optimization, it overcomes the inherent noise of user-generated content.

While the method's performance is currently capped by the quality of the initial oversegmentation, future work could integrate more advanced segmentation techniques. Additionally, the cross-modal refinement framework holds promise for other domains, such as video action recognition and large-scale image retrieval.

Summary updated

Abstract

This paper presents a cross-modal data refinement algorithm for social image parsing, or segmenting all the objects within a social image and then identifying their categories. Different from the traditional fully supervised image parsing that takes pixel-level labels as strong supervisory information, our social image parsing is initially provided with the noisy tags of images (i.e. image-level labels), which are shared by social users. By over-segmenting each image into multiple regions, we formulate social image parsing as a cross-modal data refinement problem over a large set of regions, where the initial labels of each region are inferred from image-level labels. Furthermore, we develop an efficient algorithm to solve such cross-modal data refinement problem. The experimental results on several benchmark datasets show the effectiveness of our algorithm. More notably, our algorithm can be considered to provide an alternative and natural way to address the challenging problem of image parsing, since image-level labels are much easier to access than pixel-level labels.
BibTeX
@inproceedings{lu2015social,
  author = {Zhiwu Lu and Xin Gao and Songfang Huang and Liwei Wang and Ji-Rong Wen},
  title = {{Social Image Parsing by Cross-Modal Data Refinement}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure