Cleaning up the ocean floor is a massive challenge for autonomous systems, primarily because deep learning models are "hungry" for labeled data that is incredibly hard to get underwater. In a new paper, researchers from Delft University of Technology propose a clever way to bypass the manual labeling grind: a fully automated, zero-annotation pipeline that uses generative AI to "hallucinate" trash into real underwater scenes.
TL;DR
The authors developed a pipeline that automatically generates labeled training data by inserting synthetic litter into real underwater backgrounds, using the insertion coordinates as instant labels. Their comparison of generative techniques reveals that while DALL-E 3 produces the most useful training samples—boosting detection performance by up to 96%—standard metrics for "image realism" are surprisingly poor predictors of how much a synthetic image actually helps a model learn.
What Problem Does This Paper Address?
Training a robot to spot a plastic bottle in murky, silt-heavy water requires thousands of annotated images. However, underwater data collection is expensive, and manual annotation is even worse. Experts have to squint through turbidity and light attenuation to draw bounding boxes around degraded, partially buried debris.
This "annotation bottleneck" means that even if we have raw footage, we can't easily use it to train better detectors. Existing fixes like simple "copy-paste" augmentation often look jarring and unrealistic, while fully synthetic images from GANs often fail to respect the complex physics of underwater light and texture.
Methods
The researchers proposed a zero-annotation pipeline. Instead of labeling existing images, they start with a real underwater background and a "blank" bounding box sampled at a random location. This box serves a dual purpose: it tells the generative model where to put the object, and it serves as the ground-truth label for the final training set.
They compared three insertion strategies within this framework:
- Copy-Paste: Taking real trash crops from other images and resizing/pasting them into the box.
- Stable Diffusion (SD) Inpainting: Using a fine-tuned latent diffusion model to generate an object inside the box based on a text prompt (e.g., "plastic bottle").
- DALL-E Image-Conditioned Generation: Using DALL-E’s API to contextually regenerate portions of the scene to integrate a specific trash type.
To ensure quality, they implemented a "Quality Gate": a baseline detector must be able to recognize the newly generated object with at least 50% confidence, or the image is discarded.
Datasets
The study utilized the SeaClear Marine Debris Dataset, which contains over 8,000 images from five geographic locations. The authors focused on five specific "low-data" domains within this set (like Jakljan and Slano in Croatia) to test how well their synthetic data could help when real labels are nearly non-existent.
Key Insights and Ideas
- The FID Paradox: In the world of generative AI, Fréchet Inception Distance (FID) is the gold standard for measuring "realism." Surprisingly, the researchers found that FID is a poor proxy for training utility. Stable Diffusion had the best (lowest) FID scores because it left the background untouched, but it failed to improve detection. DALL-E had much worse FID scores because it altered the surrounding pixels to blend the object in, yet it was the only method that significantly helped the detector.
- Object-Level vs. Distribution-Level Realism: Detectors don't care about global image statistics; they care about the "thingness" of the object. DALL-E excelled because it captured the specific textures of underwater decay—corrosion, marine growth, and realistic scaling—that the other methods missed.
- The 25% Rule: Training on only synthetic data resulted in near-zero performance. However, mixing a small amount of real data (25%) with the synthetic data "grounded" the model, allowing it to successfully transfer what it learned from the AI-generated trash to real-world debris.
Key Results
The quantitative results, summarized in the table below, show that DALL-E was the clear winner across almost all domains.
- Massive Gains: In the "S2" domain (Slano II), the DALL-E pipeline improved mean Average Precision ([email protected]) from 0.14 to 0.27—a relative gain of 96%.
- Failure of Open-Source Inpainting: Despite fine-tuning, Stable Diffusion inpainting failed to provide any meaningful boost over the baseline. The authors attribute this to SD's tendency to generate objects that were too large or lacked the specific "weathered" look of real underwater litter.
Limitations and Future Work
The primary limitation is that DALL-E is a "black box" proprietary model. Researchers can't see exactly how it processes prompts or what data it was trained on, which limits interpretability and reproducibility compared to open-source models like Stable Diffusion.
Moving forward, the authors suggest integrating this pipeline into a semi-autonomous loop where the system uses active learning to identify which types of trash it's struggling with and then automatically generates synthetic examples of those specific objects to "self-correct" its training.
Summary updated