Fine-grained Foreground Retrieval via Teacher-Student Learning

Zongze Wu, Dani Lischinski, Eli Shechtman

Introduction

Traditional image retrieval systems often operate at a "coarse-grained" level, identifying general categories of objects suitable for a scene. However, "fine-grained" retrieval—selecting a specific object that matches the precise style, geometry, and perspective of a background—remains a significant challenge. The primary obstacle is the lack of large-scale, annotated datasets that rank thousands of foreground objects against diverse backgrounds. Furthermore, many existing methods rely on high-quality instance segmentation masks, which are difficult to obtain for "in-the-wild" imagery.

This paper presents a self-supervised framework that leverages a Teacher-Student learning paradigm to address these issues. By learning to "hallucinate" the features of a missing object from its surrounding environment, the model eliminates the need for expensive manual labeling.

The Teacher-Student Framework

The core of the proposed method is a self-supervised domain adaptation task. The system decomposes an image into a foreground object and its corresponding background to train the model.

Overview of the Teacher-Student learning framework for fine-grained foreground retrieval.

As shown in the architecture overview, the system consists of two main components:

  1. The Teachers: Pre-trained networks act as domain experts. MarrNet extracts 3D shape and viewpoint information (such as azimuth and elevation), while VGG captures "style" features, including color and texture, through channel-wise statistics.
  2. The Student: A central network trained to predict the features the Teachers would extract from an original object, given only the background with a "hole" (bounding box).

During inference, the Student analyzes a new background and predicts the ideal features for a missing foreground element. The system then queries a library of objects, retrieving those whose Teacher-extracted features most closely align with the Student's predictions.

Saliency-Guided Feature Extraction

A key innovation of this work is the move away from rigid instance segmentation. Instead of requiring pixel-perfect masks, which are often unavailable or noisy in real-world datasets, the authors utilize Grad-CAM to generate saliency masks.

Comparison of saliency maps and masks against instance masks for feature extraction.

These saliency masks allow the style teacher to focus on the relevant object features while ignoring background noise. This approach provides the flexibility needed to handle diverse, uncurated datasets while maintaining the precision required for fine-grained matching without the overhead of manual segmentation.

Experimental Results

To evaluate the framework, the authors curated a dataset of 16.7K indoor scenes and 31K furniture objects. Performance was measured using Mean Average Precision (mAP) and Normalized Discounted Cumulative Gain (nDCG) against human-ranked ground truths.

The proposed "Full" model achieved an overall mAP of 0.500, significantly outperforming the baseline UFO (Unconstrained Foreground Object search) model, which scored 0.397. Qualitatively, the model demonstrates a superior ability to match the orientation and perspective of furniture. While baseline methods often suggest objects facing the wrong direction, this framework consistently retrieves items that integrate naturally into the room's geometry.

Conclusion and Future Work

The Teacher-Student approach effectively bypasses the data bottleneck in fine-grained retrieval by treating the background as a proxy for the missing object. While currently focused on indoor furniture, the modular nature of the "Teachers" allows the system to be adapted to other domains by swapping or adding specialized feature extractors. Future research may address the limitations of domain specificity and the reliance on the quality of pre-trained teachers to further improve retrieval accuracy.

Summary updated

Abstract

Foreground image retrieval is a challenging computer vision task. Given a background scene image with a bounding box indicating a target location, the goal is to retrieve a set of images of foreground objects from a given category, which are semantically compatible with the background. We formulate foreground retrieval as a self-supervised domain adaptation task, where the source domain consists of foreground images and the target domain of background images. Specifically, given pretrained object feature extraction networks that serve as teachers, we train a student network to infer compatible foreground features from background images. Thus, foregrounds and backgrounds are effectively mapped into a common feature space, enabling retrieval of the foregrounds that are closest to the target background in that space. A notable feature of our approach is that our training strategy does not require instance segmentation, unlike current state-of-the-art methods. Thus, our method may be applied to diverse foreground categories and background scene types and enables us to retrieve the foreground in a fine-grained manner, which is closer to the requirements of real world applications.
BibTeX
@inproceedings{wu2021fine,
  author = {Zongze Wu and Dani Lischinski and Eli Shechtman},
  title = {{Fine-grained Foreground Retrieval via Teacher-Student Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00369},
}
Figure