Redundancy-Aware Pretraining of Vision-Language Foundation Models in Remote Sensing

Mathis Jürgen Adler, Leonard Hackel, Gencer Sumbul, Begüm Demir

Abstract

Vision-Language Models (VLMs) in Remote Sensing (RS) often face computational inefficiencies due to redundant captions associated with single images. This paper introduces Weighted Feature Aggregation (WFA), a strategy that merges multiple caption features into a single representative vector. By prioritizing unique information through non-parametric uniqueness or learning-based attention, WFA reduces computational costs by nearly 80% while significantly improving zero-shot retrieval performance across standard benchmarks.

Introduction

In Remote Sensing (RS), training Vision-Language Models involves aligning satellite imagery with textual descriptions. A significant challenge in existing RS datasets (e.g., RSICD, RSITMD) is the presence of multiple, often redundant, captions for a single image. The conventional "Replication" strategy treats each image-caption pair as a distinct sample, forcing the model to process the same heavy image multiple times. This leads to substantial computational waste without necessarily providing new semantic information. This research addresses this bottleneck by proposing a method to exploit complementary information across captions efficiently.

Methods: Weighted Feature Aggregation (WFA)

The core contribution is the Weighted Feature Aggregation (WFA) strategy. Instead of replicating images, the framework extracts features from all $M$ captions associated with an image and aggregates them into a single text feature vector ($H_C$) using importance weights ($\alpha$).

The WFA architecture for pretraining VLMs with multiple captions.

The authors propose two methods for calculating these weights:

  1. Non-Parametric Uniqueness: Uses BLEU-4 scores to measure caption similarity offline. Captions with more unique details receive higher weights, adding zero training overhead.
  2. Learning-based Attention: Employs a Transformer-based attention mechanism to dynamically learn the importance of each caption during training.

The aggregated text features are aligned with image features using a contrastive loss (NT-Xent), ensuring the model learns a robust joint representation.

Results and Discussion

The WFA strategy was evaluated on zero-shot retrieval tasks using the UCM and Sydney datasets after pretraining on a large-scale combined RS dataset.

Performance comparison of different aggregation strategies across benchmarks.

Key findings include:

  • Efficiency Gains: WFA (Attention) reduces computational requirements from 171.92B FLOPs (Replication) to just 36.47B FLOPs—a nearly 5x improvement.
  • Performance Superiority: WFA (Attention) outperformed the Replication baseline by 32.4% in mAP@5 on the UCM dataset and 65.3% on the Sydney dataset.
  • Weighting Effectiveness: Simple strategies like Mean Feature or Random Selection underperform compared to WFA, demonstrating that emphasizing unique information is critical for learning rich representations.

Conclusion

Weighted Feature Aggregation provides a highly efficient and effective alternative to standard VLM pretraining in remote sensing. By mitigating the costs of data redundancy, it enables the development of powerful foundation models with significantly fewer resources. Future work will explore the applicability of WFA to other downstream tasks like Visual Question Answering and its scalability to even larger datasets.

Summary updated

Abstract

The development of foundation models through pre- training of vision-language models (VLMs) has recently attracted great attention in remote sensing (RS). VLM pretraining aims to learn image and language alignments from a large number of image-text pairs. Each pretraining image is often associated with multiple captions with redundant information due to re- peated or semantically similar phrases that result in increased pretraining and inference time of VLMs. To overcome this, we introduce a weighted feature aggregation (WFA) strategy for VLM pretraining in RS. Our strategy aims to extract and exploit complementary information from multiple captions per image, while reducing redundancies through feature aggregation with importance weighting. To calculate adaptive importance weights for different captions of each image, we propose two different techniques: i) non-parametric uniqueness; and ii) learning-based attention. In the first technique, importance weights are calcu- lated based on the bilingual evaluation understudy (BLEU)-scores of the captions to emphasize unique sentences while removing the influence of repetitive sentences. In the second technique, importance weights are learned through an attention mechanism instead of relying on hand-crafted features. The effectiveness of the proposed WFA strategy with the two techniques is analyzed in terms of downstream performance on text-to-image retrieval in RS. Experimental results show that the proposed strategy enables efficient and effective pretraining of VLMs in RS. Based on the experimental analysis, we derive guidelines for the proper se- lection of techniques, considering downstream task requirements and resource constraints. The code of this work is publicly avail- able at https://git.tu-berlin.de/rsim/redundacy-aware-rs-vlm.
BibTeX
@inproceedings{adler2025redundancy,
  author = {Mathis Jürgen Adler and Leonard Hackel and Gencer Sumbul and Begüm Demir},
  title = {{Redundancy-Aware Pretraining of Vision-Language Foundation Models in Remote Sensing}},
  booktitle = {IEEE IGARSS},
  year = {2025},
  url = {https://openreview.net/forum?id=5562jKII0i},
}
Figure