Interactive Object Detection for Tiny Objects in Large Remotely Sensed Images

Marvin Burges, Sebastian Zambanini, Robert SablatnigWACV 2025

TL;DR

This paper introduces IRTDETR, an interactive object detection system designed to efficiently label tiny objects in massive remote sensing images. By combining human feedback with a modified RTDETR architecture, it significantly speeds up annotation and reduces user effort while maintaining high accuracy, addressing a critical bottleneck in creating datasets for these challenging images.

What Problem Does This Paper Address?

Creating large-scale datasets for deep learning models is crucial, but labeling data, especially for numerous tiny objects in vast remote sensing images, is incredibly labor-intensive and costly. While automated methods are improving, they often fall short of the 100% recall and near-perfect precision required for critical applications like detecting unexploded ordnance or space debris. Existing interactive object detection methods struggle with the scale and the specific challenges of tiny objects in high-resolution imagery, often requiring significant computational resources and time. This paper aims to bridge this gap by developing an interactive system that is both efficient and accurate for this specific domain.

Methods

The core of IRTDETR is a modification of the RTDETR architecture. Key innovations include:

  • Interactive Pipeline: The system allows users to provide "User Interaction Tokens" (UITs) in the form of points or bounding boxes. These UITs are then used to guide the detection process.
  • Cross-Attention Mechanism: A cross-attention mechanism is employed between the image features and the UITs. This allows the model to learn relationships between user input and image content, effectively identifying similar objects.
  • Attention Reweighting: A module reweights the image features based on the attention maps derived from the cross-attention, focusing the model's attention on relevant areas.
  • Attention Score Similarity Loss (ASSL): A novel loss function is introduced to align predicted bounding box scores with attention-based similarity scores, ensuring that high-confidence detections are located in areas with high user-input similarity.
  • Iterative Refinement: The system supports iterative refinement, where user feedback (accepting/rejecting predictions) is used as new positive/negative samples for subsequent detection rounds, improving accuracy over time.
  • Resource Conservation: Portions of the network outputs are reused during iterative refinements to conserve computational resources, crucial for handling large images.

Datasets

The paper evaluates IRTDETR on five datasets:

  • Tiny-DOTA: Specifically designed for tiny objects in aerial images.
  • CHAI: Contains craters in historical aerial images.
  • AITOD: Another dataset for tiny objects in aerial imagery.
  • SARDET: Features objects of varying sizes in Synthetic Aperture Radar (SAR) imagery.
  • COCO: Used to evaluate performance on larger objects and to test the method's generalizability.

The paper also mentions the use of a "COCO mini train variant" for training on larger objects and the original validation set for testing.

Key Insights and Ideas

  • Human-in-the-Loop is Key for Extreme Cases: For tasks demanding near-perfect recall and precision with tiny objects in massive images, fully automated methods are insufficient. Interactive systems are essential.
  • Cross-Attention Bridges User Input and Image Features: The cross-attention mechanism effectively translates sparse user input (clicks) into actionable guidance for the object detector, even for small objects.
  • Attention-Based Loss Aligns Model with User Intent: The ASSL loss is crucial for ensuring that the model not only detects objects but also prioritizes detections in areas that align with the user's provided examples, going beyond simple location matching.
  • Efficiency is Paramount for Large Images: The design choices, like reusing network outputs and the modified architecture, are critical for making interactive annotation feasible on images that can be tens of thousands of pixels wide.

Key Results

  • Annotation Speedup: The user study demonstrated that IRTDETR is 2.46 times faster than fully manual annotation.
  • Reduced Task Load: Participants reported a 28% reduction in task load (NASA-TLX score) when using IRTDETR compared to manual annotation.
  • High Annotation Quality: The method maintains comparable annotation quality to manual methods, achieving high recall with minimal user input.
  • Superiority over C3Det: IRTDETR consistently outperformed C3Det on CHAI, AITOD, and SARDET datasets, particularly in terms of annotation quality and performance plateauing.
  • Scalability: IRTDETR can process significantly larger images (up to 6,400x6,400 pixels) compared to C3Det, which hits memory limits at 3,200x3,200 pixels.
  • Performance with Limited Data: The model shows strong performance even with as few as one user example.

Limitations and Future Work

  • Reliance on Precise Annotations: IRTDETR's performance can be affected by noisy or imprecise user labels.
  • Novel Object Detection: The model may struggle to detect entirely novel objects not represented in the training data or user input.
  • Large Object/Many Object Latency: For scenes with many large objects, the generation of numerous UITs can increase memory usage and latency.
  • Future Work:
    • Integrating Scale-MAE as a backbone for class-agnostic detection.
  • Adding active learning for dynamic class inclusion.
  • Adapting IRTDETR for oriented bounding box prediction, common in remote sensing.
  • Exploring max pooling for UITs to mitigate issues with large bounding boxes, potentially with retraining.

Summary updated

BibTeX
@inproceedings{burges2025interactive,
  author = {Marvin Burges and Sebastian Zambanini and Robert Sablatnig},
  title = {{Interactive Object Detection for Tiny Objects in Large Remotely Sensed Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure