Feature discovery and visualization of robot mission data using convolutional autoencoders and Bayesian nonparametric topic models

Genevieve Flaspohler, Nicholas Roy, Yogesh A. Girdhar

TL;DR

This paper introduces a novel hybrid approach combining Convolutional Autoencoders (CAEs) with Bayesian nonparametric topic models to automatically discover meaningful features and thematic structures within robot-collected image data. This method significantly improves the unsupervised characterization of seafloor terrains compared to traditional methods, paving the way for more intelligent robotic exploration and mission summarization.

What Problem Does This Paper Address?

The ever-increasing volume of data collected by autonomous robots, particularly from visual sensors, presents a significant challenge for analysis. Traditional methods for uncovering patterns in this data often rely on hand-crafted features or struggle to capture the inherent spatial and temporal structure of image streams. This gap between data collection and analysis hinders robots' ability to understand their environment semantically, which is crucial for adaptive exploration and effective mission summarization. The paper aims to bridge this gap by developing an unsupervised framework that can automatically learn relevant features from raw image data and discover coherent thematic topics within it.

Methods

The core of the proposed solution is a hybrid model that integrates two key components:

  • Convolutional Autoencoders (CAEs): CAEs are used to learn low-dimensional, abstract feature representations directly from images. Unlike traditional methods that rely on hand-crafted features like SIFT or SURF, CAEs can capture spatial correlations within the image data. The CAE architecture used here consists of an encoder that compresses the image into a bottleneck layer (Latent Channel Activations or LCA) and a decoder that reconstructs the original image. The LCA, representing the learned features, is then used as input for the topic model.
  • Bayesian Nonparametric Topic Models (HDP-ROST): The paper adapts a spatio-temporal Hierarchical Dirichlet Process (HDP) topic model, specifically the ROST model, to work with the features extracted by the CAE. This model is designed to uncover latent thematic structures (topics) within a sequence of data. By using the CAE-generated features, the HDP model can discover visual topics that are more semantically meaningful and directly related to visual phenomena, overcoming the limitations of the "bag-of-words" assumption in traditional topic models.

The process involves training the CAE on image data to extract LCA features, clustering these features to create a visual vocabulary, and then feeding this vocabulary into the HDP-ROST model for topic discovery.

Datasets

The proposed model was evaluated on two real-world datasets collected by an autonomous underwater vehicle (AUV) during marine robot missions:

  • Mission I: Consists of 1,117 images sampled every four seconds, featuring diverse seafloor terrains like water column, rocky seafloor, and sandy seafloor. This mission was used to test the model's ability to discover visually distinct terrain types.
  • Mission II: Comprises 2,296 images sampled similarly, primarily featuring sandy seafloor with occasional biologically interesting phenomena like crab congregations and seafloor carnage. This mission was used to test both terrain discovery and anomaly detection capabilities.

The paper also mentions that the models were evaluated on datasets that an individual biologist or data scientist could collect, highlighting the practical applicability of the approach to smaller, domain-specific datasets.

Key Insights and Ideas

  • Bridging the gap between deep learning features and probabilistic topic models: The paper successfully demonstrates how to integrate powerful deep learning feature extractors (CAEs) with established probabilistic models (HDP) to achieve a more comprehensive understanding of image data.
  • CAE features enable more semantically meaningful topics: By learning features that inherently capture spatial information, the CAE-driven topics are shown to correspond more directly to human-understandable visual phenomena (e.g., specific terrain types) compared to topics derived from traditional, hand-crafted features.
  • Unsupervised discovery of structure in challenging environments: The work highlights the potential of this hybrid approach for applications in difficult-to-access environments like the deep sea, where manual annotation is costly and time-consuming.
  • Anomaly detection benefits from probabilistic modeling: While the CAE itself is not probabilistic, its integration into a probabilistic framework allows for anomaly detection through measures like perplexity, which quantifies how well an image is explained by the learned model.

Key Results

  • Seafloor Terrain Discovery (Mission I): The hybrid HDP-CAE model achieved a mutual information score of 0.535 with hand-annotated terrain labels, significantly outperforming the standard HDP model with SURF/ORB features (0.185). This indicates that the discovered topics are highly predictive of the ground-truth seafloor terrains.
  • Anomaly Detection (Mission II): While the hybrid model showed promise in identifying anomalies, the standard HDP model with traditional features achieved a higher mutual information score (0.153) with annotated anomalies compared to the hybrid model (0.006). However, the hybrid model demonstrated less of a perplexity spike for biologically interesting images, suggesting a more robust representation of anomalies.
  • Qualitative Improvements: Visualizations show that the hybrid HDP-CAE model defines topics that correspond more directly to useful visual phenomena, such as distinct terrain types, compared to the standard HDP model.

Limitations and Future Work

  • Anomaly Detection on Imbalanced Data: The paper acknowledges that the hybrid model struggled with anomaly detection in Mission II, hypothesizing this is due to the highly imbalanced nature of the dataset (many more "normal" sandy seafloor images than anomalous ones). The CAE may not have learned sufficient features for rare anomalous events.
  • Real-time Streaming Training: The current CAE training was done offline on complete datasets. Future work aims to adapt the convolutional model for efficient, life-long training in real-time streaming applications, which is crucial for autonomous robots.
  • Improving Anomaly Detection: The authors suggest exploring methods for dealing with imbalanced datasets and using image reconstruction error directly as a metric of feature quality to improve anomaly detection.
  • Spatial Localization of Anomalies: The ability of CAEs to localize anomalies within an image is noted as a valuable future direction for robot scene understanding and real-time planning.

Summary updated

BibTeX
@inproceedings{flaspohler2017feature,
  author = {Genevieve Flaspohler and Nicholas Roy and Yogesh A. Girdhar},
  title = {{Feature discovery and visualization of robot mission data using convolutional autoencoders and Bayesian nonparametric topic models}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2017},
  doi = {10.1109/iros.2017.8202130},
}

Loading related content...

Figure