Cross-View Open-Vocabulary Object Detection in Aerial Imagery

Jyoti Kini, Rohit Gupta, Mubarak ShahSubmission to ICLR 2026 · Rejected

TL;DR

This paper introduces a cross-view contrastive learning framework that adapts vision-language object detectors (such as OWLv2) from everyday ground-level photos to overhead aerial imagery. By enforcing image-to-image alignment between aerial and ground views and associating aerial objects with expanded "bags" of descriptive text, the model achieves state-of-the-art zero-shot detection on unseen aerial benchmarks—often outperforming fully fine-tuned, closed-set detectors without suffering catastrophic forgetting.

What Problem Does This Paper Address?

Large Vision-Language Models (VLMs) like CLIP and detector variants like OWLv2 excel at Open-Vocabulary Object Detection (OVD)—detecting arbitrary novel categories described in natural language. However, their representations are overwhelmingly trained on ground-level photos.

Applying these models to aerial or satellite imagery exposes a severe domain and viewpoint gap:

  • Objects appear at extreme scale variations and arbitrary orientations.
  • Bird's-eye views cause severe geometric and perspective distortions.
  • Naive fine-tuning on aerial data causes catastrophic forgetting, wiping out the rich, open-world semantic knowledge learned from web-scale ground imagery.
  • Existing remote-sensing adaptation methods often demand extensive labeled aerial annotations, which are scarce and expensive.

The authors set out to bridge this ground-to-aerial domain shift while retaining general open-vocabulary detection capabilities.

Methods

The authors propose a structured cross-view alignment pipeline that trains an aerial vision encoder to align with a frozen ground-view encoder and language embeddings:

  1. Aerial–Ground Object Detection Correspondence:
  2. Direct Correspondence: Pairs overlapping categories between aerial imagery (xView) and ground imagery (LVIS) using ground-truth bounding boxes.
  3. Inferred Correspondence: For aerial categories missing in ground-view datasets, the ground-view detector (OWLv2) generates pseudo-bounding boxes on unannotated ground datasets (LVIS and CC12M), refined using non-maximum suppression (NMS) and confidence filtering.
  4. Aerial–Text Vocabulary Expansion:
  5. Uses an LLM (ChatGPT) to generate 4–9 synonyms, fine-grained variants, and colloquial terms per category (e.g., expanding Small Aircraft into light airplane, Cessna-type aircraft, propeller plane), creating a multi-instance "text bag."
  6. Dual Contrastive Objectives:
  7. Cross-View Image-to-Image Contrastive Loss ($\mathcal{L}_{\text{ImageA-ImageG}}$): Pulls aerial crop features toward corresponding ground crop features while keeping the ground-view encoder frozen to anchor the feature space.
  8. Multi-Instance Text Contrastive Loss ($\mathcal{L}_{\text{ImageA-Text}}$): Extends MIL-NCE to contrast aerial object features against a positive pool of synonym descriptions in its text bag versus negative terms from other categories.
  9. Standard Open-Vocabulary Detection Head: Optimizes bipartite matching, bounding box regression, and cross-entropy classification jointly with the contrastive losses.

During zero-shot inference, only the adapted aerial vision encoder and the text encoder are deployed.

Datasets

  • Training & Correspondence Source: xView (aerial dataset) aligned with LVIS and CC12M (ground-view datasets), yielding 310,548 positive correspondence pairs and expanding 60 aerial categories into 360 textual variants.
  • Evaluation Benchmarks (Zero-Shot): Evaluated without any training exposure on DOTAv2, VisDrone (both image and video splits), DIOR, and HRRSD.
  • Ground-View Retention Benchmark: Evaluated on full MS COCO to measure catastrophic forgetting.

Key Insights and Ideas

  • Anchoring to Frozen Ground Embeddings Prevents Forgetting: Naively fine-tuning OWLv2 on aerial images drops ground-view COCO mAP from 56.0 down to 32.8 (−23.2). Keeping the ground encoder fixed and pulling aerial features toward it recovers performance to 53.1 mAP (+20.3 recovery).
  • Cross-View Visual Alignment Beats Pure Text Alignment: Ablation experiments reveal that cross-view image-to-image contrastive learning contributes significantly more to downstream performance than aerial-to-text alignment alone (e.g., 25.06 mAP vs. 19.98 mAP on DOTAv2), though their combination yields the highest accuracy (27.10 mAP).
  • Better Generalization Than Aerial Foundation Models: While aerial-specific models like RemoteCLIP perform well on their source dataset (xView), their representations overfit and struggle to generalize zero-shot to other aerial sensors and viewpoints. Cross-view alignment transfers much more reliably across varied aerial domains.
  • Model-Agnostic Design: The framework is not limited to OWLv2; applying it to GroundingDINO boosted zero-shot performance on VisDrone from 33.00 to 47.15 mAP (+14.15).

Key Results

  • Outperforming Fully Supervised Baselines: In zero-shot transfer, the proposed approach surpassed dataset-specific, fully fine-tuned YOLOv11 across multiple aerial datasets:
  • DOTAv2: 38.60 mAP (vs. 32.28 for YOLOv11, +6.32)
  • VisDrone Images: 44.97 mAP (vs. 40.81 for YOLOv11, +4.16)
  • HRRSD: 74.12 mAP (vs. 70.66 for YOLOv11, +3.46)
  • DIOR: 63.91 mAP (vs. 61.68 for YOLOv11, +2.23)
  • State-of-the-Art in Aerial OVD: On VisDrone (Images), the model achieved 44.9 mAP, outperforming recent open-vocabulary detectors including GLIPv2 (36.8), GroundingDINO 1.5 (40.1), YOLO-World V2.1 (42.7), and CastDet (40.5).

Limitations and Future Work

  • Bounding Box Orientation: Evaluated using derived horizontal bounding boxes (enclosing axis-aligned rectangles) rather than oriented bounding boxes (OBBs), which are standard for densely packed, rotated aerial targets.
  • Video Temporal Dynamics: Evaluates VisDrone videos frame-by-frame without temporal smoothing or tracking across video sequences.
  • Pseudo-Label Noise: Inferred ground-view detections from OWLv2 have an initial precision of ~55.8% (at a 0.3 confidence threshold), requiring thresholding (0.45 boosts precision to 70%) to avoid training on false correspondences.
  • Inference Latency: Relying on transformer vision backbones yields modest speeds (5.2 FPS on ViT-L/14, 10.8 FPS on ViT-B/16 using an A100 GPU), which may limit deployment on real-time edge devices or UAV payloads.

Summary updated

Abstract

Traditional object detection models are typically trained on a fixed set of classes, limiting their flexibility and making it costly to incorporate new categories. Open-vocabulary object detection addresses this limitation by enabling models to identify unseen classes without explicit training. Leveraging pretrained models contrastively trained on abundantly available ground-view image-text classification pairs provides a strong foundation for open-vocabulary object detection in aerial imagery. Domain shifts, viewpoint variations, and extreme scale differences make direct knowledge transfer across domains ineffective, requiring specialized adaptation strategies. In this paper, we propose a novel framework for adapting open-vocabulary representations from ground-view images to solve object detection in aerial imagery through structured domain alignment. The method introduces contrastive image-to-image alignment to enhance the similarity between aerial and ground-view embeddings and employs multi-instance vocabulary associations to align aerial images with text embeddings. Extensive experiments on the xView, DOTAv2, VisDrone, DIOR, and HRRSD datasets are used to validate our approach. Our open-vocabulary model achieves improvements of +6.32 mAP on DOTAv2, +4.16 mAP on VisDrone (Images), and +3.46 mAP on HRRSD in the zero-shot setting when compared to finetuned closed-vocabulary dataset-specific model performance, thus paving the way for more flexible and scalable object detection systems in aerial applications.
BibTeX
@misc{kini2025cross,
  author = {Jyoti Kini and Rohit Gupta and Mubarak Shah},
  title = {{Cross-View Open-Vocabulary Object Detection in Aerial Imagery}},
  year = {2025},
  eprint = {2510.03858},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.03858},
}
Figure