Introduction
Remote Sensing Composed Image Retrieval (RSCIR) is a specialized task where users search for target satellite imagery using a "composed" query consisting of a reference image and a text instruction (e.g., "a satellite image of this airport, but with red logos in the background"). Despite its potential, the field has been hindered by two primary bottlenecks: a lack of comprehensive benchmarks and the "text-only trap" in existing zero-shot methods.
Traditional zero-shot approaches often project the query image into a text-like format, performing the search using only the text modality. This process frequently discards fine-grained visual details from the original image that the text instruction does not explicitly mention. To address these issues, researchers have introduced PatternCIR, a new benchmark, and TisCIR, a novel retrieval framework that harmonizes image and text features.
The PatternCIR Benchmark
The existing standard for remote sensing retrieval, Patterncom, provides image attributes but lacks the full query sentences and paired triplets (query image, text, target image) required for modern retrieval models. To bridge this gap, the authors developed ZS-QTG (Zero-Shot Query Text Generator).
ZS-QTG utilizes a constrained language model (CBART) to expand raw attributes into fluent, descriptive sentences. By incorporating visual guidance from RemoteCLIP, the generator ensures that the resulting text accurately reflects the visual content of the target images. The resulting PatternCIR benchmark contains 17,700 triplets organized into subsets of similar images, specifically designed to test fine-grained retrieval capabilities.
The TisCIR Framework
The proposed TisCIR (Text-image Sequential Training) method moves away from the text-only paradigm by maintaining both modalities throughout the retrieval process. It employs a two-stage sequential training strategy to ensure modality harmony.
Stage I: Multiple Self-Masking Projection (MSMP)
Instead of a generic projector, MSMP trains specialized modules for different "masked" positions within a sentence template (e.g., "A satellite image of [attribute] [class] with [background]"). This allows the model to capture diverse semantic nuances more accurately by understanding exactly which part of the query is being modified.
Stage II: Fine-Grained Image Attention (FGIA)
The FGIA module is designed to resolve conflicts between the query image and the text instruction. For instance, if the text specifies adding an object, the FGIA learns to identify and filter out irrelevant parts of the query image (like an empty runway) while retaining essential context (like surrounding terrain).
During the retrieval phase, TisCIR calculates a weighted distance using both the text embedding ($E_T^{StageIII}$) and the filtered image embedding ($E_{FG}^{StageIII}$) simultaneously. This dual-modality approach ensures that the model leverages the strengths of both inputs.
Key Results and Insights
TisCIR demonstrates a significant performance leap over existing state-of-the-art (SOTA) methods like Pic2Word and SEARLE:
- Performance Gains: TisCIR outperformed previous methods by 12.40% to 62.03% across various metrics on the PatternCIR benchmark.
- Precision: On the R@1 metric, TisCIR achieved 7.34%, a notable improvement over the next best competitor's 6.53%.
- Conflict Resolution: Qualitative analysis via heatmaps confirms that the FGIA module successfully "masks out" objects in the query image that the text instructions intended to remove or change.
Conclusion
The introduction of PatternCIR and TisCIR marks a significant advancement in remote sensing retrieval. By proving that keeping the original image in the loop—when filtered correctly—leads to higher precision, the authors have provided a new roadmap for zero-shot composed image retrieval. Future work may focus on reducing the computational overhead of dual-modality retrieval and exploring more diverse, human-generated query datasets.
Summary updated