A patch-based architecture for multi-label classification from single label annotations

Warren Jouanneau, Aurélie Bugeau, Marc Palyart, Nicolas Papadakis, Laurent VézardarXiv 2022 · OpenReview

Abstract

In this paper, we propose a patch-based architecture for multi-label classification problems where only a single positive label is observed in images of the dataset. Our contributions are twofold. First, we introduce a light patch architecture based on the attention mechanism. Next, leveraging on patch embedding self-similarities, we provide a novel strategy for estimating negative examples and deal with positive and unlabeled learning problems. Experiments demonstrate that our architecture can be trained from scratch, whereas pre-training on similar databases is required for related methods from the literature.

Introduction

Supervised multi-label classification (MLC) is a cornerstone of computer vision, yet the cost of exhaustive annotation remains a significant barrier. Standard MLC requires annotators to tag every object in an image; missing a single label (e.g., failing to tag "wood" in a scene containing both "wood" and "fabric") can lead a model to incorrectly learn that the visual features of the missing label are negative examples.

To mitigate these costs, researchers have turned to Positive and Unlabeled (PU) learning. This paper addresses the extreme case of PU learning: single-label annotation for multi-label images. The challenge lies in the "absence of evidence"—determining how to train a model to recognize multiple objects when it is unknown which other objects are present or, more importantly, which are absent.

Proposed Methodology

The authors introduce a two-pronged solution consisting of a specialized architecture and a novel "weak negative" loss function.

1. Patch-Based Architecture

Rather than processing the entire image as a single global embedding, the model decomposes the input to isolate features:

  • Multi-resolution Patch Extraction: The image is downsampled and sliced into patches of varying sizes to capture both fine-grained details and coarse context.
  • Shared CNN Backbone: Patches are processed through a lightweight CNN (EfficientNet blocks) with shared weights across all patches.
  • Attention-based Pooling: The model maintains "label codebooks" (representative embeddings for each class). A cross-attention mechanism determines which patches match specific label codebooks.
  • Shared Classifier: This component produces final predictions for each label based on the aggregated patch information.

2. Weak Negative Estimation

The core innovation is the strategy for handling missing labels. The authors hypothesize that a sufficiently small image patch likely contains only one label. If the model identifies a known label (e.g., "zebra") in certain patches, it calculates the cosine similarity between the embeddings of the known label and other unknown labels. If an unknown label (e.g., "grass") has an embedding highly similar to the "zebra" patches in that specific image, the model assumes they are likely the same entity and treats the unknown label as a weak negative. This prevents the model from over-predicting every label for every image and provides a pseudo-supervisory signal for unlabeled classes.

Experimental Results

The approach was evaluated on modified versions of the MS COCO (2014) and Pascal VOC (2012) benchmarks, where only one ground-truth label was randomly selected per image during training to simulate a weak annotation environment.

Efficiency and Performance

  • Parameter Efficiency: The proposed model uses only 250,000 parameters, a 100x reduction compared to the 23 million parameters in a standard ResNet-50.
  • Training from Scratch: On Pascal VOC, the patch-based model achieved 60.4 mAP with single-label supervision, nearly matching its full-supervision score of 61.6. Unlike standard architectures, this model does not require pre-training on massive datasets like ImageNet to achieve high performance.
  • Convergence: The architecture generalizes rapidly, requiring only 25 epochs to reach high performance, whereas competing methods often require over 100 epochs and extensive pre-training.

Key Insights

  • Patch-level Granularity: For multi-label tasks, patches are more informative than image-level embeddings because they naturally isolate objects, preventing the "feature soup" effect of global pooling.
  • Implicit Localization: The attention mechanism provides "free" heatmaps, allowing the model to localize objects (e.g., an airplane or a zebra) without explicit bounding box supervision.
  • Self-Similarity as Supervision: Using the similarity between label embeddings allows the model to make informed guesses about which labels are absent, effectively solving the PU learning problem without human intervention.

Conclusion and Future Work

This research demonstrates that a lightweight, patch-based architecture can effectively handle the challenges of single-label MLC. By leveraging patch self-similarity and attention, the model achieves state-of-the-art results with a fraction of the parameters of traditional models. Future work may explore the "one label per patch" assumption in highly cluttered scenes and the potential for batch-level negative estimation to further refine the "weak negative" signal.

Summary updated

Abstract

In this paper, we propose a patch-based architecture for multi-label classification problems where only a single positive label is observed in images of the dataset. Our contributions are twofold. First, we introduce a light patch architecture based on the attention mechanism. Next, leveraging on patch embedding self-similarities, we provide a novel strategy for estimating negative examples and deal with positive and unlabeled learning problems. Experiments demonstrate that our architecture can be trained from scratch, whereas pre-training on similar databases is required for related methods from the literature.
BibTeX
@misc{jouanneau2022patch,
  author = {Warren Jouanneau and Aurélie Bugeau and Marc Palyart and Nicolas Papadakis and Laurent Vézard},
  title = {{A patch-based architecture for multi-label classification from single label annotations}},
  year = {2022},
  eprint = {2209.06530},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.06530},
}
Figure