Handling Uncertain Tags in Visual Recognition

Arash Vahdat, Greg MoriICCV 2013 · OpenReview

Abstract: Gathering accurate training data for recognizing a set of attributes or tags on images or videos is a challenge. Obtaining labels via manual effort or from weakly-supervised data typically results in noisy training labels. We develop the FlipSVM, a novel algorithm for handling these noisy, structured labels. The FlipSVM models label noise by "flipping" labels on training examples. We show empirically that the FlipSVM is effective on images-and-attributes and video tagging datasets.

Introduction

In modern computer vision, the goal has shifted from simple object detection to a deeper understanding of visual scenes through intermediate descriptions known as tags or attributes. These attributes—such as "wearing a hat," "running," or "has wing"—provide a rich vocabulary for complex recognition tasks. However, obtaining clean, reliable data for these tags is a significant bottleneck. Human annotations are often subjective, and automated methods that scrape tags from social media are frequently incomplete or incorrect.

Standard machine learning models, such as Structural SVMs, typically assume that training labels are 100% accurate, which leads to overfitting on noise. Conversely, Latent SVMs treat tags as entirely hidden variables, ignoring the valuable, albeit imperfect, information provided by noisy training data. This paper introduces FlipSVM (FSVM), a framework designed to bridge this gap by treating training labels as uncertain hints rather than absolute truths.

Examples of noisy training labels for attributes like 'Hand' and 'Leg' across different images. Incorrect labels are marked with a red cross.

The FlipSVM Framework

FlipSVM is a max-margin approach that explicitly models label noise by allowing the algorithm to "flip" (correct) training labels during the learning process. To ensure the model remains grounded in the data, a "flip cost" ($\gamma$) is applied to every change, penalizing excessive deviations from the original labels.

Structured Interaction and Optimization

Attributes are rarely independent; for instance, the presence of a "wing" is highly correlated with a "tail." FlipSVM leverages these relationships using a tree-structured tag interaction graph. This allows the model to use contextual evidence to decide if a label should be flipped. If visual features and surrounding tags strongly suggest a specific attribute that contradicts the provided label, the model can choose to flip it for better global consistency.

The FlipSVM graphical model illustrating the relationship between training labels, flipped labels, and the most violated labeling during optimization.

The training process involves a complex "min-max" optimization problem. The goal is to find model parameters that minimize overall error while simultaneously identifying the "most violated" labeling. To maintain computational efficiency, the authors employ a heuristic involving loss-augmented inference and dynamic programming.

Experimental Evaluation

The researchers evaluated FlipSVM on two distinct domains: image attributes (a-Pascal and a-Yahoo datasets) and video event detection (TRECVID MED11).

Key Results

  • Robustness to Noise: On the a-Pascal dataset with 33% synthetic noise, FlipSVM achieved a 39.53% mean Average Precision (mAP), significantly outperforming standard SVMs (37.05%) and Latent SVMs (36.19%).
  • Video Event Detection: In the TRECVID DEV-T dataset, FlipSVM outperformed baselines across nearly every category. For example, in the "Landing Fish" category, FlipSVM reached 18.94% AP, compared to 15.69% for the next best baseline.
  • Label Quality: Analysis confirmed that the model's "flips" were highly accurate. On a-Pascal, the algorithm's flips achieved 73.30% precision, meaning nearly three out of four label changes correctly reverted a noisy label to its true state.

Conclusion and Future Work

FlipSVM demonstrates that allowing a model to selectively doubt its training data can lead to more robust visual classifiers. By modeling the structure of tag interactions and penalizing label changes, the framework effectively filters noise while retaining the useful signal in the data.

Future research directions include extending the model to handle more complex, loopy interaction graphs and refining the handling of "spam" tags (false positives) in large-scale video datasets. While the current min-max optimization is more intensive than standard methods, its ability to maintain performance in the face of significant label noise makes it a valuable tool for modern visual recognition tasks.

Summary updated

Abstract

Gathering accurate training data for recognizing a set of attributes or tags on images or videos is a challenge. Obtaining labels via manual effort or from weakly-supervised data typically results in noisy training labels. We develop the FlipSVM, a novel algorithm for handling these noisy, structured labels. The FlipSVM models label noise by "flipping" labels on training examples. We show empirically that the FlipSVM is effective on images-and-attributes and video tagging datasets.
BibTeX
@inproceedings{vahdat2013handling,
  author = {Arash Vahdat and Greg Mori},
  title = {{Handling Uncertain Tags in Visual Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2013},
  url = {https://openreview.net/forum?id=H14fYgfOWr},
}
Figure