TL;DR
This paper introduces a cross-view contrastive learning framework that adapts vision-language object detectors (such as OWLv2) from everyday ground-level photos to overhead aerial imagery. By enforcing image-to-image alignment between aerial and ground views and associating aerial objects with expanded "bags" of descriptive text, the model achieves state-of-the-art zero-shot detection on unseen aerial benchmarks—often outperforming fully fine-tuned, closed-set detectors without suffering catastrophic forgetting.
What Problem Does This Paper Address?
Large Vision-Language Models (VLMs) like CLIP and detector variants like OWLv2 excel at Open-Vocabulary Object Detection (OVD)—detecting arbitrary novel categories described in natural language. However, their representations are overwhelmingly trained on ground-level photos.
Applying these models to aerial or satellite imagery exposes a severe domain and viewpoint gap:
- Objects appear at extreme scale variations and arbitrary orientations.
- Bird's-eye views cause severe geometric and perspective distortions.
- Naive fine-tuning on aerial data causes catastrophic forgetting, wiping out the rich, open-world semantic knowledge learned from web-scale ground imagery.
- Existing remote-sensing adaptation methods often demand extensive labeled aerial annotations, which are scarce and expensive.
The authors set out to bridge this ground-to-aerial domain shift while retaining general open-vocabulary detection capabilities.
Methods
The authors propose a structured cross-view alignment pipeline that trains an aerial vision encoder to align with a frozen ground-view encoder and language embeddings:
- Aerial–Ground Object Detection Correspondence:
- Direct Correspondence: Pairs overlapping categories between aerial imagery (xView) and ground imagery (LVIS) using ground-truth bounding boxes.
- Inferred Correspondence: For aerial categories missing in ground-view datasets, the ground-view detector (OWLv2) generates pseudo-bounding boxes on unannotated ground datasets (LVIS and CC12M), refined using non-maximum suppression (NMS) and confidence filtering.
- Aerial–Text Vocabulary Expansion:
- Uses an LLM (ChatGPT) to generate 4–9 synonyms, fine-grained variants, and colloquial terms per category (e.g., expanding Small Aircraft into light airplane, Cessna-type aircraft, propeller plane), creating a multi-instance "text bag."
- Dual Contrastive Objectives:
- Cross-View Image-to-Image Contrastive Loss ($\mathcal{L}_{\text{ImageA-ImageG}}$): Pulls aerial crop features toward corresponding ground crop features while keeping the ground-view encoder frozen to anchor the feature space.
- Multi-Instance Text Contrastive Loss ($\mathcal{L}_{\text{ImageA-Text}}$): Extends MIL-NCE to contrast aerial object features against a positive pool of synonym descriptions in its text bag versus negative terms from other categories.
- Standard Open-Vocabulary Detection Head: Optimizes bipartite matching, bounding box regression, and cross-entropy classification jointly with the contrastive losses.
During zero-shot inference, only the adapted aerial vision encoder and the text encoder are deployed.
Datasets
- Training & Correspondence Source: xView (aerial dataset) aligned with LVIS and CC12M (ground-view datasets), yielding 310,548 positive correspondence pairs and expanding 60 aerial categories into 360 textual variants.
- Evaluation Benchmarks (Zero-Shot): Evaluated without any training exposure on DOTAv2, VisDrone (both image and video splits), DIOR, and HRRSD.
- Ground-View Retention Benchmark: Evaluated on full MS COCO to measure catastrophic forgetting.
Key Insights and Ideas
- Anchoring to Frozen Ground Embeddings Prevents Forgetting: Naively fine-tuning OWLv2 on aerial images drops ground-view COCO mAP from 56.0 down to 32.8 (−23.2). Keeping the ground encoder fixed and pulling aerial features toward it recovers performance to 53.1 mAP (+20.3 recovery).
- Cross-View Visual Alignment Beats Pure Text Alignment: Ablation experiments reveal that cross-view image-to-image contrastive learning contributes significantly more to downstream performance than aerial-to-text alignment alone (e.g., 25.06 mAP vs. 19.98 mAP on DOTAv2), though their combination yields the highest accuracy (27.10 mAP).
- Better Generalization Than Aerial Foundation Models: While aerial-specific models like RemoteCLIP perform well on their source dataset (xView), their representations overfit and struggle to generalize zero-shot to other aerial sensors and viewpoints. Cross-view alignment transfers much more reliably across varied aerial domains.
- Model-Agnostic Design: The framework is not limited to OWLv2; applying it to GroundingDINO boosted zero-shot performance on VisDrone from 33.00 to 47.15 mAP (+14.15).
Key Results
- Outperforming Fully Supervised Baselines: In zero-shot transfer, the proposed approach surpassed dataset-specific, fully fine-tuned YOLOv11 across multiple aerial datasets:
- DOTAv2: 38.60 mAP (vs. 32.28 for YOLOv11, +6.32)
- VisDrone Images: 44.97 mAP (vs. 40.81 for YOLOv11, +4.16)
- HRRSD: 74.12 mAP (vs. 70.66 for YOLOv11, +3.46)
- DIOR: 63.91 mAP (vs. 61.68 for YOLOv11, +2.23)
- State-of-the-Art in Aerial OVD: On VisDrone (Images), the model achieved 44.9 mAP, outperforming recent open-vocabulary detectors including GLIPv2 (36.8), GroundingDINO 1.5 (40.1), YOLO-World V2.1 (42.7), and CastDet (40.5).
Limitations and Future Work
- Bounding Box Orientation: Evaluated using derived horizontal bounding boxes (enclosing axis-aligned rectangles) rather than oriented bounding boxes (OBBs), which are standard for densely packed, rotated aerial targets.
- Video Temporal Dynamics: Evaluates VisDrone videos frame-by-frame without temporal smoothing or tracking across video sequences.
- Pseudo-Label Noise: Inferred ground-view detections from OWLv2 have an initial precision of ~55.8% (at a 0.3 confidence threshold), requiring thresholding (0.45 boosts precision to 70%) to avoid training on false correspondences.
- Inference Latency: Relying on transformer vision backbones yields modest speeds (5.2 FPS on ViT-L/14, 10.8 FPS on ViT-B/16 using an A100 GPU), which may limit deployment on real-time edge devices or UAV payloads.
Summary updated