Abstract
Vision-Language Models (VLMs) in Remote Sensing (RS) often face computational inefficiencies due to redundant captions associated with single images. This paper introduces Weighted Feature Aggregation (WFA), a strategy that merges multiple caption features into a single representative vector. By prioritizing unique information through non-parametric uniqueness or learning-based attention, WFA reduces computational costs by nearly 80% while significantly improving zero-shot retrieval performance across standard benchmarks.
Introduction
In Remote Sensing (RS), training Vision-Language Models involves aligning satellite imagery with textual descriptions. A significant challenge in existing RS datasets (e.g., RSICD, RSITMD) is the presence of multiple, often redundant, captions for a single image. The conventional "Replication" strategy treats each image-caption pair as a distinct sample, forcing the model to process the same heavy image multiple times. This leads to substantial computational waste without necessarily providing new semantic information. This research addresses this bottleneck by proposing a method to exploit complementary information across captions efficiently.
Methods: Weighted Feature Aggregation (WFA)
The core contribution is the Weighted Feature Aggregation (WFA) strategy. Instead of replicating images, the framework extracts features from all $M$ captions associated with an image and aggregates them into a single text feature vector ($H_C$) using importance weights ($\alpha$).
The authors propose two methods for calculating these weights:
- Non-Parametric Uniqueness: Uses BLEU-4 scores to measure caption similarity offline. Captions with more unique details receive higher weights, adding zero training overhead.
- Learning-based Attention: Employs a Transformer-based attention mechanism to dynamically learn the importance of each caption during training.
The aggregated text features are aligned with image features using a contrastive loss (NT-Xent), ensuring the model learns a robust joint representation.
Results and Discussion
The WFA strategy was evaluated on zero-shot retrieval tasks using the UCM and Sydney datasets after pretraining on a large-scale combined RS dataset.
Key findings include:
- Efficiency Gains: WFA (Attention) reduces computational requirements from 171.92B FLOPs (Replication) to just 36.47B FLOPs—a nearly 5x improvement.
- Performance Superiority: WFA (Attention) outperformed the Replication baseline by 32.4% in mAP@5 on the UCM dataset and 65.3% on the Sydney dataset.
- Weighting Effectiveness: Simple strategies like Mean Feature or Random Selection underperform compared to WFA, demonstrating that emphasizing unique information is critical for learning rich representations.
Conclusion
Weighted Feature Aggregation provides a highly efficient and effective alternative to standard VLM pretraining in remote sensing. By mitigating the costs of data redundancy, it enables the development of powerful foundation models with significantly fewer resources. Future work will explore the applicability of WFA to other downstream tasks like Visual Question Answering and its scalability to even larger datasets.
Summary updated