Contrastive Learning of Semantic Concepts for Open-set Cross-domain Retrieval

Aishwarya Agarwal, Srikrishna Karanam, Balaji Vasan Srinivasan, Biplab Banerjee

Abstract

Open-set cross-domain retrieval (OSCDR) presents a significant challenge in computer vision, requiring models to retrieve images from unseen categories across different artistic domains. Traditional methods relying on global features often fail when style shifts are extreme. This paper proposes the Semantic Contrastive Concept Network (SCNNet), which decomposes images into a set of orthonormal visual primitives and utilizes natural language to maintain relative semantic distances. Evaluated on benchmarks like DomainNet, SCNNet achieves a substantial 42.6% improvement in precision, demonstrating robust generalization to novel objects and styles.

Introduction

The task of image retrieval is increasingly moving towards open-set and cross-domain scenarios. A robust system should be able to identify a "bracelet" (an unseen class) in a "quick-draw" style (an unseen domain) using a photographic query. Standard retrieval systems typically struggle with this "double-blind" challenge due to domain shift and the inability to represent categories not present in the training set. Existing approaches often utilize global image features, which are sensitive to stylistic changes, or data augmentation techniques like "mixup" that can lead to manifold intrusion and model confusion.

Overview of SCNNet's approach to domain-agnostic local features and improved retrieval.

Methodology

SCNNet addresses these issues through two core components: the Concept Extraction Module (CEM) and the Semantic Ordinal Module (SOM).

  1. Concept Extraction Module (CEM): Instead of a single global vector, CEM learns a set of orthonormal basis vectors representing "visual concepts" or primitives (e.g., "roundness" or "wheels"). By applying Singular Value Decomposition (SVD) to ensure orthogonality, the model creates a distinct set of features. Any unseen class can then be represented as a linear combination of these learned concepts, allowing the model to describe a "unicycle" using the "wheel" concept learned from "cars."
  2. Semantic Ordinal Module (SOM): To bridge the gap between domains, SCNNet leverages natural language. Rather than absolute alignment between image and text embeddings, SOM enforces ordinal distance constraints. This ensures that if two categories are semantically similar in text space (e.g., "dog" and "wolf"), their visual representations remain relatively close across all domains.
The SCNNet architecture: from cross-domain triplet encoding to semantic ordinal constraints.

Experiments and Results

The framework was evaluated on three major benchmarks: DomainNet, PACS, and Sketchy Extended. In each case, the model was trained on a subset of domains and classes and tested on entirely held-out data. SCNNet significantly outperformed previous state-of-the-art methods, notably achieving a 42.6% improvement in Prec@200 on the DomainNet benchmark. Qualitative analysis via t-SNE visualizations confirmed that SCNNet successfully clusters images of unseen classes even when they originate from different, unseen domains.

Conclusion

SCNNet demonstrates that focusing on local visual primitives and relative semantic structures is key to solving open-set cross-domain retrieval. By moving away from brittle global features and absolute text-image alignment, the model achieves superior generalization. Future work will explore more complex local relationships and reducing the computational cost of the SVD-based concept extraction.

Summary updated

Abstract

We consider the problem of image retrieval where query images during testing belong to classes and domains both unseen during training. This requires learning a feature space that has the ability to generalize across both classes and domains together. To this end, we propose semantic contrastive concept network (SCNNet), a new learning framework that helps take a step towards class and domain generalization in a principled fashion. Unlike existing methods that rely on global object representations, SCNNet proposes to learn local feature vectors to facilitate unseen-class generalization. To this end, SCNNet’s key innovations include (a) a novel trainable local concept extraction module that learns an orthonormal set of basis vectors, and (b) computes local features for any unseen-class data as a linear combination of the learned basis set. Next, to enable unseen-domain generalization, SCNNet proposes to generate supervisory signals from an adjacent data modality, i.e., natural language, by mining freely available textual label information associated with images. SCNNet derives these signals from our novel trainable semantic ordinal distance constraints that ensure semantic consistency between pairs of images sampled from different domains. Both the proposed modules above enable end-to-end training of the SC-NNet, resulting in a model that helps establish state-of-the-art performance on the standard DomainNet, PACS, and Sketchy benchmark datasets with average Prec@200 improvements of 42.6%, 6.5%, and 13.6% respectively over the most recently reported results.
BibTeX
@inproceedings{agarwal2023contrastive,
  author = {Aishwarya Agarwal and Srikrishna Karanam and Balaji Vasan Srinivasan and Biplab Banerjee},
  title = {{Contrastive Learning of Semantic Concepts for Open-set Cross-domain Retrieval}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00410},
}
Figure