Dynamic Weighted Combiner for Mixed-Modal Image Retrieval

Fuxiang Huang, Lei Zhang, Xiaowei Fu, Suqi SongAAAI 2024 · DOI

TL;DR

This paper introduces a "Dynamic Weighted Combiner" (DWC) for mixed-modal image retrieval (MMIR), addressing the limitations of existing methods that overlook modality importance disparity and noisy web data. DWC dynamically adjusts the contribution of image and text queries, purifies features to mitigate noise, and leverages CLIP for better cross-modal understanding, achieving state-of-the-art results.

What Problem Does This Paper Address?

Mixed-modal image retrieval (MMIR) aims to find relevant images using a combination of an image and text query. While this offers a more intuitive search experience than single-modal retrieval, current MMIR methods struggle with two critical issues:

  1. Modality Importance Disparity: The relative importance of the image and text components in a query varies significantly across different search intents and real-world scenarios. Previous methods often treat them equally or rely too heavily on one modality, leading to poor performance when the dominant modality doesn't align with the user's true intent. For instance, a user might provide an image of a shirt but primarily intend to search for a specific style of pants described in text, making the shirt image less relevant.
  2. Noisy Web Data and Label Bias: Datasets scraped from the web often contain noisy text descriptions and inherent biases. Users describe the same object in diverse ways, and labeling errors are common. This noise can lead to models overfitting to incorrect information, hindering their generalization capabilities.

Methods

The proposed Dynamic Weighted Combiner (DWC) tackles these challenges with a multi-component architecture:

  • Editable Modality De-equalizer (EMD): This is the core innovation for handling modality importance disparity. Instead of simple concatenation or direct weighting of features, EMD first purifies the image and text features. It uses cross-modal attention mechanisms (spatial attention for images, word attention for text) to highlight relevant parts of each modality based on the other. This is followed by a global attention refinement. Finally, an adaptive weighting module dynamically combines these purified features, assigning weights based on their estimated importance for the specific query.
  • Dynamic Soft-Similarity Label Generator (SSG): To combat noisy labels and data bias, SSG generates "soft" similarity labels instead of relying on hard, binary (match/no-match) labels. This allows the model to learn from partially relevant "negative" samples, effectively leveraging more information from the dataset and reducing overfitting to noisy ground truth.
  • CLIP-based Mutual Enhancement: To bridge modality gaps and improve cross-modal understanding, DWC incorporates a pre-trained CLIP model. This model acts as a powerful feature extractor and is used in a "mutual enhancement" strategy. The two streams (one using traditional CNN/LSTM, the other using CLIP) are trained alternately, sharing knowledge through a KL-divergence distillation loss. This encourages both streams to learn robust, aligned representations.
  • Mixed-Modality Contrastive Loss: This loss function is used to facilitate similarity learning by encouraging the combined query representation to be similar to the target image representation, guided by the soft-similarity labels.

Datasets

The paper evaluates the proposed DWC on three real-world datasets:

  • Fashion200K: A dataset for fashion image retrieval.
  • Shoes: Another dataset focused on shoe retrieval.
  • FashionIQ: A dataset designed for text-conditioned image retrieval.

The authors do not introduce new datasets but utilize existing benchmarks to demonstrate the effectiveness of their approach.

Key Insights and Ideas

  • Modality Importance is Dynamic: The paper strongly emphasizes that the contribution of image vs. text in a query isn't fixed. A one-size-fits-all approach to combining features is suboptimal.
  • Feature Purification is Crucial: Simply combining raw features or applying static weights can amplify noise. Purifying features before weighting is key to robust performance.
  • Soft Labels Unlock More Data: Moving beyond hard labels to soft similarity labels is a powerful way to leverage the inherent ambiguity and richness of web-scale datasets, improving model robustness.
  • CLIP as a Strong Foundation: Leveraging pre-trained large vision-language models like CLIP significantly aids in bridging modality gaps and improving cross-modal alignment.
  • Mutual Learning Enhances Robustness: Training two distinct streams (CNN/LSTM and CLIP) in a mutually enhancing manner leads to more robust and generalizable representations.

Key Results

The DWC model consistently outperforms state-of-the-art methods across the evaluated datasets.

  • FashionIQ: DWC achieves a performance improvement of up to 12.49% on R@1 and 7.4% on average compared to the second-best method.
  • Shoes: DWC shows improvements of 0.83% on R@1 and 1.06% on average.
  • Fashion200K: DWC demonstrates compelling results, with improvements of 0.25% on R@10, 3.64% on R@10 for shirts, and 1.98% on R@10 for dresses compared to the second-best method.
  • Qualitative results (shown in Figures 6, 7, and 8) demonstrate the model's ability to retrieve relevant images based on nuanced mixed-modal queries. Attention visualizations (Figure 9) confirm the effectiveness of the spatial and word attention mechanisms.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the use of CLIP and multiple streams suggests a higher computational cost compared to simpler models. The paper mentions using lightweight global attention for the CLIP stream to reduce computation.
  • Comparison with More Complex Models: On FashionIQ, DWC is slightly lower than FashionVLP in R@50, which is noted as a more complex model with auxiliary modules like object detection. This suggests potential avenues for further integration of specialized modules.
  • Future Work: The paper's conclusion highlights the effectiveness of the proposed components. Future work could explore further refinements of the EMD and SSG modules, investigate different pre-trained models, or apply the DWC framework to other cross-modal retrieval tasks. The authors also mention that supplementary material contains further experimental analyses, hinting at deeper explorations not fully detailed in the main paper.

Summary updated

BibTeX
@inproceedings{huang2024dynamic,
  author = {Fuxiang Huang and Lei Zhang and Xiaowei Fu and Suqi Song},
  title = {{Dynamic Weighted Combiner for Mixed-Modal Image Retrieval}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i3.28004},
}

Loading related content...

Figure