Popeye: A Unified Visual-Language Model for Multi-Source Ship Detection from Remote Sensing Imagery

Wei Zhang, Miaoxin Cai, Tong Zhang, Guoqiang Lei, Yin Zhuang, Xuerui Mao

Abstract

Ship detection in remote sensing (RS) imagery is challenged by sensor diversity, extreme scale variations, and complex backgrounds. We present Popeye, a unified Vision-Language Model (VLM) that addresses these challenges by integrating multi-scale visual features with the reasoning capabilities of Large Language Models (LLMs). Popeye supports both optical and Synthetic Aperture Radar (SAR) imagery and can perform horizontal (HBB) and oriented bounding box (OBB) detection via a conversational interface. Our model demonstrates superior zero-shot performance across multiple benchmarks, bridging the gap between specialized detection models and general-purpose VLMs.

Introduction

Detecting ships from satellite imagery is a critical yet difficult task. Unlike standard photography, remote sensing data is captured from a nadir perspective, resulting in small targets, significant scale variations, and interference from maritime environments like docks and waves. Historically, the field has been fragmented by sensor types (Optical vs. SAR) and task formats (Horizontal vs. Oriented Bounding Boxes).

Popeye supports multi-source imagery and diverse detection tasks through a unified conversational interface.

Popeye overcomes these barriers by treating ship detection as a language-guided reasoning task. By converting coordinate data into a uniform text-based "instruction-answer" format, Popeye can switch between detection modes and sensor types seamlessly based on user prompts. This approach allows the model to leverage the broad knowledge of foundation models while specializing in the nuances of maritime remote sensing.

Methodology

Popeye is built upon the LLaMA-7B foundation and incorporates several architectural innovations to handle the nuances of remote sensing data.

Hybrid Experts Encoder

To capture both high-level semantics and fine-grained structural details, Popeye utilizes a dual-backbone approach. It combines a CLIP encoder for semantic understanding with a DINOv2 encoder for structural precision. Images are processed at multiple resolutions to ensure that even the smallest vessels are detected accurately.

The Popeye architecture integrates a Hybrid Experts Encoder with an LLM and instruction adapters for joint training.

Unified Labeling and Training

The model employs a unified labeling paradigm where various annotation formats (HBB, OBB) are translated into text sequences. Training occurs in two stages:

  1. Visual-Language Alignment: Initial training on natural scene data (e.g., COCO) to establish basic image-to-text logic.
  2. Instruction Adaptation: Fine-tuning on ship-specific data using Low-Rank Adaptation (LoRA) and a specialized instruction adaptor that adds lightweight learnable parameters to the Transformer layers.

Furthermore, Popeye can be integrated with the Segment Anything Model (SAM). By providing bounding box prompts generated by Popeye, the system enables high-fidelity ship segmentation without requiring explicit mask-based training.

The MMShip Dataset

To support this unified approach, the authors introduced MMShip, a comprehensive benchmark containing 81,000 instruction-following samples. MMShip unifies four major datasets: DOSR and DOTA for optical imagery, and SSDD and HRSID for SAR imagery. This dataset allows the model to learn cross-domain features that generalize well to unseen data, proving that pre-training on natural scenes enhances reasoning for remote sensing objects.

Experimental Results

Popeye achieves state-of-the-art zero-shot performance, outperforming both general-purpose VLMs and specialized detection models in many scenarios.

  • Horizontal Detection: On the ShipRSimagenet dataset, Popeye reached 56.68 AP@40, significantly higher than GroundingDINO (35.13) and Sphinx (41.61).
  • SAR Imagery: The model showed exceptional strength in radar data, achieving 75.59 AP@40 on the DSSDD dataset, nearly doubling the performance of the next best VLM, Sphinx (35.04).
  • Oriented Detection: Despite being a generalist, Popeye achieved 58.15 AP@40 on the HRSC2016 dataset, rivaling specialized OBB models like S2A-Net.

Conclusion and Future Work

Popeye represents a significant step toward a "universal" observer for maritime monitoring. By leveraging the flexibility of LLMs, it unifies disparate sensors and tasks into a single, intuitive framework. Future developments will focus on expanding the model's capabilities to include fine-grained ship classification (e.g., identifying specific vessel types), supporting infrared imagery, and improving visual grounding for complex textual descriptions.

Summary updated

BibTeX
@article{zhang2024popeye,
  author = {Wei Zhang and Miaoxin Cai and Tong Zhang and Guoqiang Lei and Yin Zhuang and Xuerui Mao},
  title = {{Popeye: A Unified Visual-Language Model for Multi-Source Ship Detection from Remote Sensing Imagery}},
  journal = {IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens.},
  year = {2024},
  doi = {10.1109/jstars.2024.3488034},
}
Figure