Language Guided Local Infiltration for Interactive Image Retrieval

Fuxiang Huang, Lei ZhangWorkshops @ CVPR 2023 · DOI · arXiv

Introduction

Interactive Image Retrieval (IIR) is the task of finding a target image based on a reference image and a natural language query describing desired modifications (e.g., "change the blue shirt to red" or "make the small gray object larger").

Traditional methods often rely on "global fusion," where image and text features are simply concatenated or added. However, this approach frequently leads to "global smearing," where unrelated parts of the image (like the background or other objects) are inadvertently altered. To address this, researchers have introduced Language Guided Local Infiltration (LGLI), a framework designed to achieve surgical precision in image editing for retrieval tasks.

Methodology

The LGLI system moves beyond simple feature fusion by implementing a locally aware architecture consisting of two primary components:

  1. Language Prompt Visual Localization (LPVL): This module identifies the specific region intended for modification. It utilizes a frozen Faster R-CNN for region proposal and a frozen CLIP model to match these regions with the "localization text" from the user's query, resulting in a precise localization mask.
  2. Text Infiltration with Local Awareness (TILA): Instead of a single fusion step, TILA modules are integrated into multiple layers of the image encoder. Using a cross-modal attention mechanism, TILA "infiltrates" text features specifically into the masked area of the visual features. This ensures that the text modifies only the relevant visual semantics while preserving the rest of the image representation.
The overall architecture of the LGLI framework, illustrating the interaction between the LPVL and TILA modules across multiple feature layers.

Experimental Setup

The authors evaluated LGLI on three diverse benchmarks:

  • CSS (Clevr-Single-Sent): A synthetic dataset for testing basic attribute modifications (color, size, presence).
  • Fashion200k: A real-world dataset focused on complex textures and fashion-specific attributes.
  • MIT-States: A challenging dataset involving noun-adjective pairs that test the model's understanding of state changes (e.g., "diced" vs. "whole").

Results and Key Insights

LGLI consistently outperformed state-of-the-art (SOTA) methods across all benchmarks. On the CSS dataset, it achieved a 93.3% R@1 accuracy, a 5.5% improvement over previous methods. On Fashion200k, it reached 26.5% R@1, surpassing the previous SOTA by 4.1%.

Qualitative analysis confirms that the model excels at "surgical" modifications. By treating text as a local modifier, LGLI avoids tampering with unrelated semantics, maintaining high fidelity for the parts of the image that should remain unchanged.

Qualitative retrieval results on the CSS dataset. The green boxes indicate the ground truth target images, demonstrating the model's ability to handle addition, removal, and attribute changes.

Limitations and Future Work

While highly effective for local attribute changes, the system's performance is tied to the accuracy of the initial localization. In complex or cluttered scenes (as seen in MIT-States), the LPVL module occasionally struggles. Furthermore, the model finds it difficult to process abstract, global modifications like "windblown," which do not correspond to a single maskable region. Future research will likely focus on balancing local infiltration with global semantic shifts to handle these more complex, atmospheric queries.

Summary updated

BibTeX
@inproceedings{huang2023language,
  author = {Fuxiang Huang and Lei Zhang},
  title = {{Language Guided Local Infiltration for Interactive Image Retrieval}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
  doi = {10.1109/cvprw59228.2023.00650},
}
Figure