FOR: Finetuning for Object Level Open Vocabulary Image Retrieval

Hila Levi, Guy Heller, Dan LeviWACV 2025

TL;DR

This paper introduces FOR, a novel framework for fine-tuning object-centric open-vocabulary image retrieval (OC-OVIR) models. By combining a specialized "SUM-CLIP" head with a multi-objective training strategy, FOR significantly boosts retrieval accuracy for unseen object categories without sacrificing efficiency, even in semi-supervised settings.

What Problem Does This Paper Address?

The core challenge is accurately retrieving images based on text descriptions of objects, especially when those objects weren't part of the model's original training data (open-vocabulary retrieval). Existing methods like CLIP excel at matching general image-caption pairs but struggle with pinpointing specific objects within an image for retrieval. While approaches like Dense-CLIP and Cluster-CLIP offer improvements by generating more localized embeddings or summarizing them, they either lack fine-tuning capabilities or require computationally expensive post-processing. This leaves a gap for models that can be adapted to specific domains, efficiently retrieve object-level information, and handle novel categories effectively.

Methods

The proposed framework, FOR (Finetuning for Object-centric Open-vocabulary Image Retrieval), introduces two key innovations:

  • SUM-CLIP Head: This is a novel variant of the CLIP's visual encoder head. Unlike CLIP's single global embedding or Dense-CLIP's dense patch embeddings, SUM-CLIP uses additional learnable queries and decoder layers. This design allows it to capture multiple objects within an image by generating a summarized set of representative embeddings, crucial for scalability and efficient retrieval. The linear layers of the attention module are often frozen to preserve CLIP's original vision-language alignment.
  • Multi-Objective Training: FOR employs a training scheme with two branches:
  • Supervised Loss: Fine-tuning on the target dataset using its base categories.
  • Pseudo-Label Loss: This branch compensates for "open-vocabulary catastrophic forgetting" by using pseudo-labels derived from external datasets (like ImageNet-21K) and a fixed CLIP-based architecture (Cluster-CLIP). This allows the model to learn from a broader range of concepts beyond the base categories.

This combination enables fine-tuning on limited labeled data while leveraging external knowledge and maintaining performance on novel categories.

Datasets

The paper evaluates FOR on three publicly available datasets:

  • COCO 2017: Used for object detection and divided into 48 base and 17 novel categories.
  • LVIS: A long-tail object recognition dataset with 1,203 semantic categories, categorized into frequent (base), common (base), and rare (novel).
  • nuImages: A dataset for autonomous driving, with 8 rare categories designated as novel and 10 frequent categories as base.

Key Insights and Ideas

  • Fine-tuning is Crucial for OC-OVIR: The paper strongly advocates for fine-tuning vision-language models for object-centric retrieval, showing that off-the-shelf models are insufficient.
  • SUM-CLIP Captures Object Diversity Efficiently: The SUM-CLIP head's ability to generate a small set of representative embeddings per image is key to balancing retrieval accuracy with scalability, outperforming methods that rely on a single embedding or many dense embeddings.
  • Pseudo-Labels Mitigate Forgetting: The use of pseudo-labels from external, diverse datasets is vital for preventing the model from forgetting novel categories during fine-tuning.
  • Stability Over Plasticity: Freezing the linear layers of the SUM-CLIP head, while allowing other parts to train, seems to be a critical factor in maintaining the model's strong zero-shot retrieval capabilities.
  • Semi-Supervised Learning is Highly Effective: FOR demonstrates impressive performance even with very limited labeled data (as low as 1%), highlighting the power of its pseudo-labeling strategy.

Key Results

  • Accuracy Improvements: FOR achieves significant gains, outperforming the state-of-the-art Cluster-CLIP by up to 7.6 mAP@50 on novel categories across the evaluated datasets. It also shows substantial improvements on base categories.
  • Efficiency: FOR eliminates the need for expensive CPU post-processing required by methods like Cluster-CLIP, leading to a three-fold reduction in visual inference time.
  • Scalability: By using a small number of embeddings per image (e.g., 50), FOR is well-suited for large-scale retrieval systems.
  • Semi-Supervised Performance: On COCO, FOR achieves comparable results to a fully supervised baseline with only 1-10% of labeled data, and shows improvements of up to 75 mAP@50 over a supervised-only baseline on novel categories.

Limitations and Future Work

  • Limitations: The paper notes that fine-tuning a standard CLIP head (without SUM-CLIP) can decrease performance on novel categories, possibly due to losing relevant pseudo-label training signals with its "one-to-one" correspondence. The effectiveness of pseudo-labels can vary depending on the overlap with target dataset categories.
  • Future Work: The authors suggest that leveraging frameworks for learning compact open-vocabulary representations holds promise for various applications like detection, segmentation, and image generation. They plan to explore integrating these with large language models.

Summary updated

BibTeX
@inproceedings{levi2025finetuning,
  author = {Hila Levi and Guy Heller and Dan Levi},
  title = {{FOR: Finetuning for Object Level Open Vocabulary Image Retrieval}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure