Guiding Labelling Effort for Efficient Learning With Georeferenced Images

Takaki Yamada, Miquel Massot-Campos, Adam Prügel-Bennett, Oscar Pizarro, Stefan B. Williams, Blair ThorntonIEEE TPAMI 2023 · DOI · OpenReview

Abstract

Training deep learning models for environmental monitoring is often bottlenecked by the massive effort required for expert manual labeling. This paper introduces a semi-supervised pipeline that uses georeference data and smart clustering to achieve 85% of the accuracy of a fully-labeled dataset using just 40 prioritized annotations—a nearly 250x reduction in human effort.

Introduction

In fields like marine biology or aerial forestry monitoring, robotic platforms such as drones and Autonomous Underwater Vehicles (AUVs) collect tens of thousands of images per mission. However, converting these images into actionable data typically requires expert scientists to manually label them—a process that is both time-consuming and expensive.

Standard AI approaches, such as fine-tuning models pre-trained on ImageNet, often underperform in these domains because seafloor or satellite imagery differs significantly from generic datasets. Furthermore, environmental datasets are frequently "unbalanced," with common classes (like sand) vastly outnumbering rare but critical features (like endangered coral). Random sampling for labeling often misses these rare classes entirely. This research addresses the challenge of training high-performing classifiers on a per-mission basis without requiring thousands of manual annotations.

The LGA-SS Pipeline

The proposed LGA-SS (Location Guided Autoencoder - Semi-Supervised) pipeline streamlines the transition from raw data collection to high-accuracy inference through four integrated stages.

Overview of the LGA-SS pipeline, illustrating the flow from georeferenced data collection to unsupervised feature extraction, prioritized labeling, and final supervised learning.

1. Unsupervised Feature Learning (LGA)

The process begins with a Location Guided Autoencoder. Unlike standard autoencoders, this model is regularized by georeference data (GPS coordinates). It operates on the principle that images captured in close spatial proximity are likely to share similar features. This spatial constraint helps the model learn a "latent representation" that is highly specific to the target environment.

2. Prioritized Labelling

To maximize the impact of human effort, the system employs hierarchical $k$-means clustering within the latent space. Instead of labeling random images, the system identifies "centroids"—the most representative images of each cluster. This ensures that even rare classes are captured and presented to the expert for annotation.

3. Pseudo-labelling

After an expert labels a minimal subset (e.g., 40 images), a Radial Basis Function Support Vector Machine (R-SVM) propagates these labels to the remaining unlabelled images. This "pseudo-labelling" step creates a larger, albeit noisier, training set from a tiny human-labeled seed.

4. CNN Training and Inference

Finally, a standard Convolutional Neural Network (such as ResNet18) is trained using a combination of high-quality human labels and the generated pseudo-labels. This hybrid dataset allows the model to achieve high performance with minimal manual intervention.

Experimental Validation

The method was validated across four distinct environmental datasets with highly skewed class distributions:

  • Seafloor: Color imagery from an AUV featuring classes like "Bacterial Mat," "Shell Fragment," and "Sand."
  • Terrestrial Aerial (Mountain, Island, Urban): Cropped images from ESRI World Imagery representing diverse ecosystems and human infrastructure.

Key Insights

  • Geography as a Self-Supervised Signal: Using spatial proximity as a regularizer allows the model to learn relevant environmental features more effectively than pixel-only analysis.
  • Diversity Over Quantity: The research demonstrates that selecting which images to label is more critical than the total volume of labels. Hierarchical clustering ensures that rare but important classes are not overlooked.
  • Mission-Specific Training: Because the pipeline is efficient enough to run on a single mission's data, it avoids the "transferability gap" where models trained in one geographic region fail in another.

Results

The LGA-SS method achieved accuracy comparable to standard transfer learning while using an order of magnitude fewer annotations. Notably, in the seafloor dataset, the model reached 85% of the accuracy of a fully-labeled model (10,000 labels) using only 40 prioritized annotations. Furthermore, the method successfully recovered rare classes like "Artificial Object" and "Bacterial Mat" with just 20 total labels, whereas random sampling failed to identify them at such low sample sizes.

Limitations and Future Work

While the pipeline significantly reduces human effort, it introduces computational overhead, as an autoencoder and CNN must be trained for every new mission. Additionally, the benefits of "Probabilistic Pseudo-labelling" to mitigate label noise were only evident at higher label counts ($M \approx 1000$). Future research will focus on integrating this workflow into real-time robotic platforms, allowing autonomous systems to request specific labels from operators during active missions.

Summary updated

Abstract

We describe a novel semi-supervised learning method that reduces the labelling effort needed to train convolutional neural networks (CNNs) when processing georeferenced imagery. This allows deep learning CNNs to be trained on a per-dataset basis, which is useful in domains where there is limited learning transferability across datasets. The method identifies representative subsets of images from an unlabelled dataset based on the latent representation of a location guided autoencoder. We assess the method's sensitivities to design options using four different ground-truthed datasets of georeferenced environmental monitoring images, where these include various scenes in aerial and seafloor imagery. Efficiency gains are achieved for all the aerial and seafloor image datasets analysed in our experiments, demonstrating the benefit of the method across application domains. Compared to CNNs of the same architecture trained using conventional transfer and active learning, the method achieves equivalent accuracy with an order of magnitude fewer annotations, and 85 % of the accuracy of CNNs trained conventionally with approximately 10,000 human annotations using just 40 prioritised annotations. The biggest gains in efficiency are seen in datasets with unbalanced class distributions and rare classes that have a relatively small number of observations.
BibTeX
@article{yamada2023guiding,
  author = {Takaki Yamada and Miquel Massot-Campos and Adam Prügel-Bennett and Oscar Pizarro and Stefan B. Williams and Blair Thornton},
  title = {{Guiding Labelling Effort for Efficient Learning With Georeferenced Images}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  doi = {10.1109/tpami.2021.3140060},
}
Figure