IDEA: An Invariant Perspective for Efficient Domain Adaptive Image Retrieval

Haixin Wang, Hao Wu, Jinan Sun, Shikun Zhang, Chong Chen, Xian-Sheng Hua, Xiao Luo

This paper introduces IDEA (Invariance-acquired Domain Adaptive Hashing), a novel framework designed to solve the "domain shift" problem in image retrieval. By explicitly disentangling causal (semantic) features from non-causal (domain-specific) features, the authors create hash codes that remain robust even when the search environment changes drastically.

TL;DR

IDEA improves cross-domain image retrieval by using a causal analysis lens to separate label-related information from domain-related noise. It employs a generative model to simulate "interventions"—swapping domain features—to force the resulting hash codes to be truly domain-invariant, significantly outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Image retrieval often relies on learning to hash, where high-dimensional images are converted into compact binary codes. This allows for lightning-fast searches using Hamming distance. However, most models assume the training data (source domain) and the query data (target domain) come from the same distribution.

In the real world, this is rarely true. A model trained on high-quality studio photos might fail when searching through grainy mobile phone uploads. This is known as domain shift. Existing "Domain Adaptive Hashing" methods try to align these domains but often fail because they entangle semantic information (like "this is a bike") with domain information (like "this was taken in the rain"). If the model thinks "rain" is part of what makes a bike a bike, retrieval fails in sunny weather.

Methods

The authors propose a three-pronged approach to achieve "Invariance-acquired" hashing:

  1. Causal Disentanglement: Using a Structural Causal Model (SCM), the authors define images as a combination of a Causal part (related to labels) and a Non-causal part (related to domain). They use an encoder and the Information Bottleneck principle to separate these features. Causal features are pushed to maximize mutual information with labels, while non-causal features are restricted from carrying label info.
  2. Consistency Learning: To ensure the hash codes are discriminative, the model enforces that similar images (those sharing labels in the source or augmented views in the target) produce similar binary codes.
  3. Invariance under Intervention: This is the most novel part. The authors train a Generative Model to reconstruct images from the causal and non-causal parts. They then perform "interventions" by swapping the non-causal features of one image with another. By forcing the hash code of these synthetic, "counterfactual" images to remain the same as the original, the model learns to ignore domain-specific noise.
The IDEA framework: from image encoding and disentanglement to generative reconstruction and intervention.

Datasets

The model was evaluated on three standard benchmark datasets for domain adaptation:

  • Office-Home: 65 object categories across four domains (Art, Clipart, Product, Real-World).
  • Office-31: 31 categories from three domains (Amazon, Webcam, DSLR).
  • Digits: Cross-domain tasks between MNIST and USPS handwritten digits.

Key Insights and Ideas

  • The Causal Lens: Instead of just trying to make source and target features "look" the same, IDEA asks why they are different. By identifying the causal path from label to image, it isolates the information that should be invariant.
  • Generative "Stress Testing": Using a generative model to create synthetic samples isn't just for data augmentation; it's a way to explicitly penalize the model for being sensitive to non-causal factors.
  • Principled Disentanglement: The use of Information Bottleneck provides a theoretical foundation for ensuring that the non-causal features don't "leak" semantic information, which would otherwise lead to "reckless" correlation-based learning.

Key Results

IDEA consistently outperformed both unsupervised and transfer hashing baselines across all datasets and code lengths (16 to 128 bits).

  • Quantitative: On the Office-Home and Office-31 datasets, IDEA achieved an average Mean Average Precision (MAP) of 57.03%, a significant jump over the previous state-of-the-art (PEACE) at 51.22%. Precision-Recall curves further demonstrate IDEA's robust retrieval performance across various domain adaptation tasks.
Precision-Recall curves across multiple domain adaptation tasks demonstrating IDEA's superior performance over baselines like PEACE and DHLing.
  • Qualitative: The authors also note that t-SNE visualizations produce much tighter, better-separated clusters for different classes compared to baselines, indicating more discriminative hash codes.

Limitations and Future Work

The authors identify a few areas for growth:

  • Open-Set Scenarios: Currently, the model assumes the source and target domains share the same set of classes. Handling "unseen" classes in the target domain remains a challenge.
  • Multimodal Retrieval: The current work focuses on image-to-image retrieval. Extending this invariant perspective to text-to-image or other multimodal tasks is a promising direction.
  • Generative Complexity: The quality of the "interventions" depends on the generative model. Using more advanced architectures (like Diffusion Models) could potentially improve the robustness of the learned invariance.

Summary updated

Abstract

In this paper, we investigate the problem of unsupervised domain adaptive hashing, which leverage knowledge from a label-rich source domain to expedite learning to hash on a label-scarce target domain. Although numerous existing approaches attempt to incorporate transfer learning techniques into deep hashing frameworks, they often neglect the essential invariance for adequate alignment between these two domains. Worse yet, these methods fail to distinguish between causal and non-causal effects embedded in images, rendering cross-domain retrieval ineffective. To address these challenges, we propose an Invariance-acquired Domain AdaptivE HAshing (IDEA) model. Our IDEA first decomposes each image into a causal feature representing label information, and a non-causal feature indicating domain information. Subsequently, we generate discriminative hash codes using causal features with consistency learning on both source and target domains. More importantly, we employ a generative model for synthetic samples to simulate the intervention of various non-causal effects, ultimately minimizing their impact on hash codes for domain invariance. Comprehensive experiments conducted on benchmark datasets validate the superior performance of our IDEA compared to a variety of competitive baselines.
BibTeX
@inproceedings{wang2023idea,
  author = {Haixin Wang and Hao Wu and Jinan Sun and Shikun Zhang and Chong Chen and Xian-Sheng Hua and Xiao Luo},
  title = {{IDEA: An Invariant Perspective for Efficient Domain Adaptive Image Retrieval}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=77i6itptQW},
}
Figure