Embedding Generalized Semantic Knowledge Into Few-Shot Remote Sensing Segmentation

Qi Wang, Qi Wang 0009, Yuyu Jia, Wei Huang, Junyu Gao, Qiang Li

TL;DR

Researchers have developed a new framework called Holistic Semantic Embedding (HSE) that significantly improves few-shot segmentation in remote sensing by incorporating textual class descriptions. By blending visual features with "general knowledge" from language models like CLIP, the system can better identify complex objects (like planes or harbors) even when they look vastly different across different images.

What Problem Does This Paper Address?

Few-Shot Segmentation (FSS) aims to segment objects in an image given only a tiny handful of annotated examples (the "support" set). In remote sensing (satellite and aerial imagery), this is notoriously difficult due to high intra-class variance.

For instance, two "planes" might look completely different depending on the sensor angle, lighting, or aircraft type. If your only support example is a small white Cessna, the model might fail to recognize a large grey military transport plane in the "query" image. Previous methods relied solely on mining visual cues from these sparse examples, which often aren't enough to build a robust representation of a category.

Methods

The authors propose the HSE (Holistic Semantic Embedding) approach, which introduces textual modality—specifically Class Description (CD) embeddings—into the visual feature extraction process.

The architecture consists of two primary novel modules:

  1. Spatial Dense Interaction (SDI) Module: Instead of just concatenating text and image data, this module uses a self-attention mechanism. It extends visual support features with CD embeddings along the spatial dimension, allowing the model to learn which spatial characteristics are most relevant to the text description (e.g., "a photo of a plane").
  2. Global Content Modulation (GCM) Module: This module operates on the channel dimension. It converts CD embeddings and visual prototypes into an "enhancement coefficient." This coefficient is used to boost the signal of channels that contain category-relevant information while suppressing background noise and interference.

By combining these, the model creates a "holistic" representation that uses the stability of language to anchor the volatility of remote sensing visuals.

Datasets

The study utilizes the iSAID-5$^i$ dataset, a popular benchmark for remote sensing FSS.

  • Composition: 15 categories (including ship, storage tank, baseball diamond, tennis court, etc.).
  • Scale: 18,076 images for training and 6,363 for testing.
  • Setup: The categories are split into three folds (5 categories each) for cross-validation, ensuring the model is tested on "novel" classes it didn't see during training.

Key Insights and Ideas

  • Language as an Anchor: Textual descriptions (like "harbor" or "large vehicle") are category-level and don't change based on the specific image. This provides a "generalized" knowledge base that compensates for the "sparse" visual info in a 1-shot or 5-shot scenario.
  • Spatial vs. Channel Synergy: The authors found that embedding semantic knowledge is most effective when done both spatially (to find where the object is based on its description) and globally via channels (to determine what features define the class).
  • Language Model Flexibility: The framework is "plug-and-play" with different language models. While CLIP performed the best (likely due to its native vision-language alignment), the system also showed strong results using SBERT and GloVe.

Key Results

The HSE method set a new state-of-the-art on the iSAID-5$^i$ benchmark:

  • ResNet50 Backbone: Achieved a mean Intersection over Union (mIoU) of 39.72% in the 1-shot setting, outperforming the previous best (HMRE) by 2.73%.
  • 5-Shot Performance: Reached 43.15% mIoU, a 2.09% improvement over prior work.
  • Qualitative Success: Visual results show that HSE is much better at reducing "false activations"—it doesn't get confused by background textures that look similar to the target object as easily as baseline models do.

Limitations and Future Work

The authors noted a "diminishing returns" effect regarding the number of support samples. As the number of visual examples increases (e.g., moving toward 10-shot), the relative benefit of the textual semantic knowledge decreases because the visual cues eventually become sufficient on their own.

Future research could look into more sophisticated ways to align these modalities or explore how to handle "ambiguous" class descriptions where a simple text prompt might not capture the full complexity of a remote sensing category.

Summary updated

BibTeX
@article{wang2025embedding,
  author = {Qi Wang and Qi Wang and Yuyu Jia and Wei Huang and Junyu Gao and Qiang Li},
  title = {{Embedding Generalized Semantic Knowledge Into Few-Shot Remote Sensing Segmentation}},
  journal = {IEEE TGRS},
  year = {2025},
  doi = {10.1109/tgrs.2024.3519772},
}
Figure