Exploiting weakly supervised visual patterns to learn from partial annotations

Kaustav Kundu, Joseph TigheNeurIPS 2020 · OpenReview

TL;DR

Training deep learning models on massive datasets like OpenImages is hindered by "partial annotations," where less than 1% of potential labels are actually verified for any given image. This paper introduces Soft Exposure (SE), a method that treats un-annotated labels not as "ignored" or "strictly negative," but as "soft" signals weighted by their visual and semantic relationships to known labels, leading to a 2-10% performance boost across major benchmarks.

The Challenge of Partial Annotations

In the world of computer vision, "more data" usually means "more problems" for annotators. As the number of visual categories grows into the thousands (e.g., OpenImages has ~9.6k labels), it becomes financially and logistically impossible to ask a human to verify every single label for every single image. To manage this, researchers use "federated datasets," where only a tiny subset of labels is checked for each image.

Examples of partially annotated images showing known positive and negative labels, illustrating the vast number of un-annotated categories.

The standard approach to training on these partially annotated datasets is to simply ignore the un-annotated labels (the "No Exposure" approach). However, this throws away a massive amount of potential supervisory signal. Conversely, assuming all un-annotated labels are negative (the "Full Exposure" approach) introduces significant noise, as many "missing" labels are actually present in the image but were simply never checked. The authors seek a middle ground: how can we use these "unknown" labels to help the model learn without being led astray by missing positives?

Soft Exposure: A Middle Ground

The authors propose a novel training strategy called Soft Exposure (SE). It builds on a surprisingly strong baseline they identified: treating all un-annotated labels as negatives but giving them less "weight" than confirmed labels.

The core of SE is a Knowledge Distillation framework using temperature scaling. Instead of a hard "0" (negative) or "1" (positive), un-annotated labels are assigned a "soft" target. The "softness" (or confidence) of this target is determined by a temperature term $T$, which is calculated based on two types of similarity:

  1. Label-to-Label Similarity ($d_L$): If an un-annotated label (e.g., "Boy") is semantically and visually close to a confirmed positive label in that image (e.g., "Man"), the model is less certain that "Boy" is a negative.
  2. Image-to-Image Similarity ($d_I$): The method looks at the $k$-nearest neighbor images in the feature space. If an image's neighbors are frequently labeled with a specific category, that category is treated with more "softness" for the current image.

By combining these distances, the model assigns a unique temperature to every un-annotated label for every image, effectively "softening" the penalty for labels that are likely to be present but unconfirmed.

Key Insights

  • The "Full Exposure" Surprise: One of the most interesting findings is that treating all un-annotated labels as negatives (Full Exposure) is actually a better baseline than ignoring them (No Exposure). Because the number of true negatives in a large label space is orders of magnitude higher than positives, the "noisy" negative signal still provides better feature representations than no signal at all.
  • Softening the Noise: While Full Exposure is a good start, it's fundamentally flawed because it penalizes the model for correctly identifying objects that weren't labeled. Soft Exposure fixes this by using the "wisdom of the crowd" (image neighbors) and semantic logic (label relationships) to lower the penalty on likely-positive un-annotated labels.
  • Impact on Rare Labels: The method is particularly effective for labels that are frequently missing or rare. By looking at relationships, the model can "hallucinate" a supervisory signal for categories that humans rarely bother to annotate.

Results across Benchmarks

The authors evaluated their approach on five diverse and challenging multi-label benchmarks, including CIFAR100, MS COCO, OpenImages v5, and LVIS. The Soft Exposure (SE) method consistently outperformed existing state-of-the-art methods.

Performance comparison showing Soft Exposure (SE) outperforming baselines across varying label proportions.
  • Performance Gain: Across all datasets, SE outperformed baselines by 2-10% in mean Average Precision (mAP) and mean F1 scores.
  • OpenImages: On the massive OpenImages benchmark, SE achieved an mAP of 80.67, significantly higher than the "Weighted No Exposure" baseline (77.18) and the "Full Exposure" baseline (76.89).
  • Convergence: The authors noted that SE models converge faster than baselines because they receive a richer, albeit softer, supervisory signal from the very beginning of training.

Limitations and Future Work

  • Feature Dependency: The similarity metrics ($d_L$ and $d_I$) rely on representations from a pre-trained ImageNet model. If the pre-trained model has biases or lacks certain visual concepts, the "softness" calculations might be inaccurate.
  • Single-Pass Training: The current work focuses on a single round of training. The authors suggest that an iterative approach—where the model's own predictions are used to further refine the "soft" labels in a second pass—could yield even higher gains.
  • Computational Overhead: While the distance calculations are done only once, the $k$-nearest neighbor search could become a bottleneck as datasets scale into the tens of millions of images.

Summary updated

Abstract

As classifications datasets progressively get larger in terms of label space and number of examples, annotating them with all labels becomes non-trivial and expensive task. For example, annotating the entire OpenImage test set can cost $6.5M. Hence, in current large-scale benchmarks such as OpenImages and LVIS, less than 1\% of the labels are annotated across all images. Standard classification models are trained in a manner where these un-annotated labels are ignored. Ignoring these un-annotated labels result in loss of supervisory signal which reduces the performance of the classification models. Instead, in this paper, we exploit relationships among images and labels to derive more supervisory signal from the un-annotated labels. We study the effectiveness of our approach across several multi-label computer vision benchmarks, such as CIFAR100, MS-COCO panoptic segmentation, OpenImage and LVIS datasets. Our approach can outperform baselines by a margin of 2-10% across all the datasets on mean average precision (mAP) and mean F1 metrics.
BibTeX
@inproceedings{kundu2020exploiting,
  author = {Kaustav Kundu and Joseph Tighe},
  title = {{Exploiting weakly supervised visual patterns to learn from partial annotations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=OZX3J4nFvi7g},
}
Figure