Leveraging Metadata in Representation Learning With Georeferenced Seafloor Imagery

Takaki Yamada, Miquel Massot-Campos, Adam Prügel-Bennett, Stefan B. Williams, Oscar Pizarro, Blair ThorntonIEEE RA-L 2021 · DOI · OpenReview

Introduction

Autonomous Underwater Vehicles (AUVs) are essential for modern oceanography, capturing vast quantities of high-resolution seafloor imagery. However, the sheer volume of data creates a bottleneck: human experts cannot label all images, and raw pixels are too computationally expensive for real-time robotic decision-making. This paper introduces a self-supervised method that leverages "free" metadata—specifically GPS coordinates and depth—to improve representation learning. By forcing a neural network to group images that are spatially or bathymetrically close, the system achieves significant boosts in classification accuracy without requiring manual labels during training.

Methodology

The core of the proposed approach is a Metadata Regularised Autoencoder. The architecture utilizes a convolutional autoencoder (based on AlexNet) to compress complex images into meaningful latent vectors. The innovation lies in a multi-part loss function:

  1. Reconstruction Loss ($L_{rec}$): Ensures the network can reconstruct the original image from its compressed representation.
  2. Horizontal Location Loss ($L_{loc}$): Regularizes the latent space based on the assumption that images taken in close proximity are likely to share visual characteristics.
  3. Depth Loss ($L_{dep}$): Incorporates the ecological reality that seafloor habitats are strongly influenced by depth.
Overview of the Metadata Regularised Autoencoder architecture, showing how affinity matrices for location and depth are used to regularize the latent representation.

To implement this, the researchers calculate "affinity matrices" for both the metadata and the latent space. By minimizing the Kullback–Leibler (KL) divergence between these matrices, the network organizes its internal representations to match the physical map of the seafloor. A contrastive mini-batch sampling strategy is also employed to ensure the network encounters a balanced mix of nearby and distant image pairs.

Dataset and Evaluation

The method was validated using the Tasmania Dataset, comprising 32,097 color seafloor images collected by the Sirius AUV. The survey covered a 1.6 × 1.7 km region with depths ranging from 28m to 96m. For evaluation purposes, 2,221 images were manually annotated into eight distinct classes, such as Kelp, High Relief Reef, and Coarse Sand.

Visualization of the Tasmania dataset, including class examples and plots showing the strong correlation between habitat types and depth/location.

Key Insights and Results

  • Metadata as Supervision: Location and depth are recorded automatically, providing a cost-free signal to guide learning. Depth, in particular, acts as a bridge, allowing the network to regularize learning across different, non-contiguous dives.
  • Performance Gains: Integrating both location and depth metadata increased image classification accuracy by an average of 10.9% across various classifiers. The largest improvement was a 15.1% boost observed with a Radial Basis Function SVM compared to a standard autoencoder.
  • Robustness: The use of heavy-tailed t-distributions in the loss function makes the model robust to noise. If specific metadata does not correlate with visual features, the network learns to ignore it, preventing performance degradation.
  • Unsupervised Summarization: Clustering these metadata-aware representations allowed for the automatic selection of representative images that accurately summarized the survey, successfully identifying rare habitats that standard methods often overlook.

Conclusion and Future Work

While highly effective, the method's performance is tied to the accuracy of the AUV's navigation suite. Future work may involve incorporating additional sensor data, such as acoustic backscatter or water temperature, to further refine representations. The resulting compact, meaningful vectors are ideal for low-latency, on-robot processing in future autonomous missions.

Summary updated

Abstract

Camera equipped Autonomous Underwater Vehicles (AUVs) are now routinely used in seafloor surveys. Obtaining effective representations from the images they collect can enable perception-aware robotic exploration such as information-gain-guided path planning and target-driven visual navigation. This letter develops a novel self-supervised representation learning method for seafloor images collected by AUVs. The method allows deep-learning convolutional autoencoders to leverage multiple sources of metadata to regularise their learning, prioritising features observed in images that can be correlated with patterns in their metadata. The impact of the proposed regularisation is examined on a dataset consisting of more than 30 k colour seafloor images gathered by an AUV off the coast of Tasmania. The metadata used to regularise learning in this dataset consists of the horizontal location and depth of the observed seafloor. The results show that including metadata in self-supervised representation learning can increase image classification accuracy by up to 15% and never degrades learning performance. We show how effective representation learning can be applied to achieve class balanced representative image identification for summarised understanding of imbalanced class distributions in an unsupervised way.
BibTeX
@article{yamada2021leveraging,
  author = {Takaki Yamada and Miquel Massot-Campos and Adam Prügel-Bennett and Stefan B. Williams and Oscar Pizarro and Blair Thornton},
  title = {{Leveraging Metadata in Representation Learning With Georeferenced Seafloor Imagery}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2021},
  doi = {10.1109/lra.2021.3101881},
}
Figure