High-Fidelity Lake Extraction Via Two-Stage Prompt Enhancement: Establishing A Novel Baseline and Benchmark

Ben Chen, Xuechao Zou, Kai Li, Yu Zhang, Junliang Xing, Pin Tao

TL;DR

Researchers have developed LEPrompter, a framework that uses a "teacher-student" approach to train lake extraction models. By using automatically generated prompts (points, boxes, and masks) during training to guide the model and then removing them during inference, they achieved state-of-the-art accuracy without any additional computational cost at deployment.

What Problem Does This Paper Address?

Extracting lake boundaries from remote sensing (satellite) imagery is notoriously difficult. Lakes come in irregular shapes, and satellite data is often "noisy" due to reflections, clouds, or complex surrounding terrain like snow and mountains.

While the Segment Anything Model (SAM) recently revolutionized interactive segmentation using prompts, it isn't a perfect fit for automated remote sensing. SAM requires manual input (like clicking a point) to work, lacks specific semantic labels (it just finds "objects"), and uses massive, computationally expensive backbones. This paper aims to bring the power of prompt-based learning to fully automated lake extraction while keeping the models lightweight and efficient.

Methods

The authors introduce LEPrompter, a two-stage prompt enhancement framework:

  1. Prompt-Based Training (The "Teacher" Stage): During the initial phase of training, the model is given "hints" in the form of prompts (points, boxes, or masks) derived from the ground truth. A lightweight Prompt Encoder and Prompt Decoder (adding only 1.23M parameters) process these hints and fuse them with the image features.
  2. Prompt-Free Training (The "Student" Stage): After a certain number of steps (50k is found to be optimal), the prompts are deactivated. The model must now perform the extraction independently, having "learned" the features the prompts were highlighting.
  3. Prompt-Free Inference: When the model is actually used on new data, the prompt modules are completely bypassed. This means the accuracy gains from training are preserved, but the model runs as fast as a standard segmentation network.

Datasets

The study utilizes two primary remote sensing datasets:

  • SW (Surface Water) Dataset: 17,596 images (256x256).
  • QTPL (Qinghai-Tibet Plateau Lake) Dataset: 6,773 images with high spatial resolution.

The Benchmark Contribution: The authors created a new benchmark by applying morphological operations (like DBSCAN clustering, dilation, and erosion) to the ground truth of these datasets. This allowed them to generate five types of automated prompts: random points, center points, bounding boxes, filled masks, and unfilled masks.

To test generalization, they also evaluated the method on medical imaging datasets (CVC-ClinicDB and ISIC2018).

Key Insights and Ideas

  • Prompts as Training Wheels: The core idea is that prompts shouldn't just be for interaction; they can serve as a superior supervisory signal. By guiding the model's attention during training, the model learns to identify intricate lake details it might otherwise ignore.
  • The "Goldilocks" Principle of Prompts: Interestingly, the researchers found that more prompts aren't always better. Providing too much information (like a perfect mask prompt) can make the model "lazy" and over-reliant on the prompt, leading to worse performance when the prompt is removed. They found that three random points provided the best balance for guiding the model.
  • Zero-Cost Accuracy: Because the prompt architecture is only active during training, this method provides a "free" performance boost for real-world applications where low latency and low power consumption are critical.

Key Results

The LEPrompter framework consistently improved the performance of various backbones (SegFormer, PoolFormer, SegNeXt, and LEFormer):

  • Accuracy Boost: On the SW dataset, LEFormer equipped with LEPrompter achieved an mIoU of 91.53%, a significant improvement over the baseline.
  • Generalization: The method worked across different domains, improving mIoU on medical polyp and skin lesion datasets by over 1% in some cases.
  • Efficiency: During the prompt-based training stage, the overhead is minimal (0.95G Flops), and during inference, the overhead is zero.

Limitations and Future Work

The authors acknowledge that while their automated prompt generation is effective, it relies on existing ground truth. In a truly "in-the-wild" scenario without any labels, generating these prompts for training remains a challenge.

Future work could explore:

  • Applying this two-stage enhancement to other remote sensing tasks like building or road extraction.
  • Investigating more complex "negative prompts" (telling the model what isn't a lake) to further reduce false positives in noisy environments.

Summary updated

Abstract

Lake extraction from remote sensing imagery is a complex challenge due to the varied lake shapes and data noise. Current methods rely on multispectral image datasets, making it challenging to learn lake features accurately from pixel arrangements. This, in turn, affects model learning and the creation of accurate segmentation masks. This paper introduces a prompt-based dataset construction approach that provides approximate lake locations using point, box, and mask prompts. We also propose a two-stage prompt enhancement framework, LEPrompter, with prompt-based and prompt-free stages during training. The prompt-based stage employs a prompt encoder to extract prior information, integrating prompt tokens and image embedding through self- and cross-attention in the prompt decoder. Prompts are deactivated to ensure independence during inference, enabling automated lake extraction without introducing additional parameters and GFlops. Extensive experiments showcase performance improvements of our proposed approach compared to the previous state- of-the-art method. The source code is available at https://github.com/BastianChen/LEPrompter.
BibTeX
@inproceedings{chen2024high,
  author = {Ben Chen and Xuechao Zou and Kai Li and Yu Zhang and Junliang Xing and Pin Tao},
  title = {{High-Fidelity Lake Extraction Via Two-Stage Prompt Enhancement: Establishing A Novel Baseline and Benchmark}},
  booktitle = {ICME},
  year = {2024},
  doi = {10.1109/icme57554.2024.10688015},
}
Figure