Abstract
The NASA Planetary Data System (PDS) hosts millions of images acquired from Mars. To help users quickly find images of interest, we have developed and deployed content-based classification and search capabilities for Mars orbital and surface images. This paper describes the process of training, evaluating, and deploying two CNN classifiers—HiRISENet and MSLNet—and reports on three years of deployment usage and lessons learned.
The Challenge of Massive Planetary Archives
The NASA PDS is a massive archive containing millions of images from Mars missions. While these images are tagged with metadata like time, instrument, and location, scientists often need to search by content—for example, "find all images containing craters" or "show me rover tracks." Manually labeling millions of images is impossible, and traditional metadata doesn't capture the visual features within the frame. This work describes the evolution of automated classification necessary to make these archives searchable by visual content.
Methodology: Transfer Learning and Calibration
The researchers employed transfer learning, taking the AlexNet architecture and fine-tuning it for Martian imagery. A significant technical focus was classifier calibration. Modern neural networks are often overconfident; to ensure the PDS Image Atlas only shows highly reliable results, the team implemented post-hoc calibration methods:
- Temperature Scaling: Rescaling model outputs using a single parameter.
- Matrix Scaling: A complex linear transformation providing per-class scaling.
- Abstention: The system is designed to "abstain" from providing a label if the posterior probability is below 0.9, prioritizing precision over recall to maintain user trust.
Labeled Datasets for Orbital and Surface Imagery
The study utilizes two high-quality labeled datasets:
- HiRISE Orbital Data Set (v3): Contains 10,815 landmark images from the Mars Reconnaissance Orbiter, categorized into 8 classes such as Crater, Dark dune, and Slope streak.
- MSL Surface Data Set (v2): Contains 2,900 images from the Curiosity rover, spanning 19 classes including geological features (Layered rock) and rover parts (Wheel, Drill hole).
Key Results and Deployment Impact
Between late 2016 and August 2020, the classifiers satisfied 62,613 searches on the PDS Image Atlas.
- HiRISENet Performance: The orbital classifier achieved 96.7% accuracy on the test set (using a 0.9 confidence threshold) with a 20% abstention rate.
- MSLNet Performance: The surface classifier achieved 90.3% accuracy at the same threshold, with a higher abstention rate (51.8%) due to the variety of surface terrain.
- Popularity: "Crater" was identified as the most frequently queried class for orbital images.
Limitations and Future Work
The current single-label assumption is a limitation, as many images contain multiple objects of interest. Future iterations aim to move toward multi-label classification. Additionally, the team is working on MERNet for the Spirit and Opportunity rover archives and investigating label shift adaptation to handle visual changes as rovers move into geologically distinct territories.
Summary updated