Abstract
Open-set cross-domain retrieval (OSCDR) presents a significant challenge in computer vision, requiring models to retrieve images from unseen categories across different artistic domains. Traditional methods relying on global features often fail when style shifts are extreme. This paper proposes the Semantic Contrastive Concept Network (SCNNet), which decomposes images into a set of orthonormal visual primitives and utilizes natural language to maintain relative semantic distances. Evaluated on benchmarks like DomainNet, SCNNet achieves a substantial 42.6% improvement in precision, demonstrating robust generalization to novel objects and styles.
Introduction
The task of image retrieval is increasingly moving towards open-set and cross-domain scenarios. A robust system should be able to identify a "bracelet" (an unseen class) in a "quick-draw" style (an unseen domain) using a photographic query. Standard retrieval systems typically struggle with this "double-blind" challenge due to domain shift and the inability to represent categories not present in the training set. Existing approaches often utilize global image features, which are sensitive to stylistic changes, or data augmentation techniques like "mixup" that can lead to manifold intrusion and model confusion.
Methodology
SCNNet addresses these issues through two core components: the Concept Extraction Module (CEM) and the Semantic Ordinal Module (SOM).
- Concept Extraction Module (CEM): Instead of a single global vector, CEM learns a set of orthonormal basis vectors representing "visual concepts" or primitives (e.g., "roundness" or "wheels"). By applying Singular Value Decomposition (SVD) to ensure orthogonality, the model creates a distinct set of features. Any unseen class can then be represented as a linear combination of these learned concepts, allowing the model to describe a "unicycle" using the "wheel" concept learned from "cars."
- Semantic Ordinal Module (SOM): To bridge the gap between domains, SCNNet leverages natural language. Rather than absolute alignment between image and text embeddings, SOM enforces ordinal distance constraints. This ensures that if two categories are semantically similar in text space (e.g., "dog" and "wolf"), their visual representations remain relatively close across all domains.
Experiments and Results
The framework was evaluated on three major benchmarks: DomainNet, PACS, and Sketchy Extended. In each case, the model was trained on a subset of domains and classes and tested on entirely held-out data. SCNNet significantly outperformed previous state-of-the-art methods, notably achieving a 42.6% improvement in Prec@200 on the DomainNet benchmark. Qualitative analysis via t-SNE visualizations confirmed that SCNNet successfully clusters images of unseen classes even when they originate from different, unseen domains.
Conclusion
SCNNet demonstrates that focusing on local visual primitives and relative semantic structures is key to solving open-set cross-domain retrieval. By moving away from brittle global features and absolute text-image alignment, the model achieves superior generalization. Future work will explore more complex local relationships and reducing the computational cost of the SVD-based concept extraction.
Summary updated