Mars Image Content Classification: Three Years of NASA Deployment and Recent Advances

Kiri Wagstaff, Steven Lu, Emily Dunkel, Kevin Grimes, Brandon Zhao, Jesse Cai, Shoshanna B. Cole, Gary Doran, Raymond Francis, Jake H. Lee, Lukas Mandrake

Abstract

The NASA Planetary Data System (PDS) hosts millions of images acquired from Mars. To help users quickly find images of interest, we have developed and deployed content-based classification and search capabilities for Mars orbital and surface images. This paper describes the process of training, evaluating, and deploying two CNN classifiers—HiRISENet and MSLNet—and reports on three years of deployment usage and lessons learned.

The Challenge of Massive Planetary Archives

The NASA PDS is a massive archive containing millions of images from Mars missions. While these images are tagged with metadata like time, instrument, and location, scientists often need to search by content—for example, "find all images containing craters" or "show me rover tracks." Manually labeling millions of images is impossible, and traditional metadata doesn't capture the visual features within the frame. This work describes the evolution of automated classification necessary to make these archives searchable by visual content.

Methodology: Transfer Learning and Calibration

The researchers employed transfer learning, taking the AlexNet architecture and fine-tuning it for Martian imagery. A significant technical focus was classifier calibration. Modern neural networks are often overconfident; to ensure the PDS Image Atlas only shows highly reliable results, the team implemented post-hoc calibration methods:

  • Temperature Scaling: Rescaling model outputs using a single parameter.
  • Matrix Scaling: A complex linear transformation providing per-class scaling.
  • Abstention: The system is designed to "abstain" from providing a label if the posterior probability is below 0.9, prioritizing precision over recall to maintain user trust.

Labeled Datasets for Orbital and Surface Imagery

The study utilizes two high-quality labeled datasets:

  1. HiRISE Orbital Data Set (v3): Contains 10,815 landmark images from the Mars Reconnaissance Orbiter, categorized into 8 classes such as Crater, Dark dune, and Slope streak.
  2. MSL Surface Data Set (v2): Contains 2,900 images from the Curiosity rover, spanning 19 classes including geological features (Layered rock) and rover parts (Wheel, Drill hole).
Example of a slope streak feature from the HiRISE orbital dataset.

Key Results and Deployment Impact

Between late 2016 and August 2020, the classifiers satisfied 62,613 searches on the PDS Image Atlas.

  • HiRISENet Performance: The orbital classifier achieved 96.7% accuracy on the test set (using a 0.9 confidence threshold) with a 20% abstention rate.
  • MSLNet Performance: The surface classifier achieved 90.3% accuracy at the same threshold, with a higher abstention rate (51.8%) due to the variety of surface terrain.
  • Popularity: "Crater" was identified as the most frequently queried class for orbital images.
A crater, the most frequently queried image class in the PDS Image Atlas.

Limitations and Future Work

The current single-label assumption is a limitation, as many images contain multiple objects of interest. Future iterations aim to move toward multi-label classification. Additionally, the team is working on MERNet for the Spirit and Opportunity rover archives and investigating label shift adaptation to handle visual changes as rovers move into geologically distinct territories.

Summary updated

Abstract

The NASA Planetary Data System hosts millions of images acquired from the planet Mars. To help users quickly find images of interest, we have developed and deployed content-based classification and search capabilities for Mars orbital and surface images. The deployed systems are publicly accessible using the PDS Image Atlas. We describe the process of training, evaluating, calibrating, and deploying updates to two CNN classifiers for images collected by Mars missions. We also report on three years of deployment including usage statistics, lessons learned, and plans for the future.
BibTeX
@inproceedings{wagstaff2021mars,
  author = {Kiri Wagstaff and Steven Lu and Emily Dunkel and Kevin Grimes and Brandon Zhao and Jesse Cai and Shoshanna B. Cole and Gary Doran and Raymond Francis and Jake H. Lee and Lukas Mandrake},
  title = {{Mars Image Content Classification: Three Years of NASA Deployment and Recent Advances}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=rYXQTgukBK},
}
Figure