ViSeR: Visual Self-Regularization

Hamid Izadinia, Pierre GarriguesWorkshops @ CVPR 2020 · DOI

Problem Statement

Deep learning models for image recognition and localization are notoriously data-hungry. While datasets like ImageNet have driven massive progress, they are often biased toward single, centered objects. More realistic datasets like MS COCO and Visual Genome—which feature multiple objects in complex, "in-context" scenes—are significantly more expensive to annotate.

There exists a massive untapped resource in billions of unlabeled images available online (e.g., the YFCC 100M dataset). However, standard regularization techniques often rely on synthetic noise (like Gaussian blur or adversarial pixel attacks) which fail to capture the natural variations found in real-world photography. ViSeR (Visual Self-Regularization) bridges this gap by using real, unlabeled images as "natural" perturbations to improve model robustness and localization.

The ViSeR Framework

The ViSeR pipeline utilizes a four-step process to leverage unlabeled data effectively:

  1. Initial Supervised Training: A Fully Convolutional Network (FCN) is trained on a labeled dataset using a Noisy-OR multiple instance learning approach. This allows the model to learn object localization even with only image-level labels.
  2. Semantic Retrieval: For every labeled training image, the model extracts a feature vector and searches the YFCC 100M dataset for "nearest neighbors"—images that are semantically similar but visually distinct.
  3. Label Propagation: Labels from the original training image are propagated to these retrieved unlabeled "Regularizer samples."
  4. Visual Self-Regularization: The model is re-trained using both original data and Regularizer samples, forcing consistent predictions across semantically related images.
The ViSeR architecture employs an FCN with Noisy-OR pooling to enable localization and classification, which serves as the foundation for semantic retrieval.

To handle the scale of 100 million images, the authors implemented an approximate distributed search algorithm using MapReduce, making the retrieval process feasible within a few hours.

Key Insights: Real-World Perturbations

A core tenet of ViSeR is that real-world images provide a more meaningful regularization signal than synthetic noise. Traditional Adversarial Training (AT) or Virtual Adversarial Training (VAT) creates "fake" perturbations. In contrast, ViSeR argues that a different photograph of a similar scene follows the natural distribution of real-world data more accurately.

Comparison of decision boundaries across different regularization techniques. ViSeR (right) demonstrates a superior ability to capture the true underlying data distribution compared to standard methods like Dropout or VAT.

Furthermore, by retrieving images based on high-level features, the system finds images with similar spatial and semantic compositions (e.g., a "dog on a couch"). This "in-context" advantage is crucial for multi-label scene understanding where the relationship between objects matters as much as the objects themselves.

Experimental Results

ViSeR demonstrated significant improvements across both classification and localization tasks:

  • Object Localization (MS COCO): Achieved 50.64% mean Average Precision (mAP), significantly outperforming previous benchmarks like ProNet (43.5%).
  • Object Classification (MS COCO): Reached 75.48% mAP, surpassing standard supervised FCNs and VAT.
  • Scalability: The method scaled effectively to the thousands of categories in Visual Genome, providing a consistent ~2% boost over non-regularized baselines.
  • Robustness: In controlled experiments with synthetic multimodal data, ViSeR proved more resilient to bias than standard dropout or adversarial training.

Conclusion and Future Work

ViSeR proves that large-scale web data can seamlessly "fill in the gaps" of human-annotated datasets by acting as natural regularizers. While the method currently depends on the quality of the initial supervised model, future work aims to incorporate user-provided metadata (tags and titles) from datasets like YFCC to further refine the retrieval and label propagation process.

Summary updated

BibTeX
@inproceedings{izadinia2020viser,
  author = {Hamid Izadinia and Pierre Garrigues},
  title = {{ViSeR: Visual Self-Regularization}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  doi = {10.1109/cvprw50498.2020.00479},
}
Figure