Researchers from NASA’s Jet Propulsion Laboratory (JPL) and Oregon State University recently presented a new approach to navigating the vast archives of Martian imagery at AAAI-2024. Their system, Proto-MSLNet, moves beyond traditional "black-box" deep learning to provide an interpretable search tool for scientists and the public. By explaining its classifications through visual "prototypes," the system allows users to understand exactly why an image is labeled as a specific feature, such as a "wheel joint" or a "drill hole."
The Challenge of Opaque Archives
The NASA Planetary Data System (PDS) hosts millions of images from decades of Mars missions. For scientists, finding specific geological features or rover components within this behemoth is a daunting manual task. While standard Convolutional Neural Networks (CNNs) can automate this search, they often fail to provide context. If a model misidentifies a feature, a scientist has no way of knowing if the error stemmed from lighting conditions, camera artifacts, or a fundamental misunderstanding of the object. This lack of transparency hinders scientific trust and prevents effective model debugging.
Methodology: Prototypical Reasoning
To address this, the team utilized a Prototypical Part Network (ProtoPNet) architecture. Unlike standard models that rely on abstract mathematical features, ProtoPNet identifies specific "prototypes"—representative patches from training images—to make decisions.
When classifying a new image, the model compares parts of the input to these learned prototypes. As shown in the figure above, the system calculates a similarity score and applies a weight to each match. This allows the model to effectively say: "I classify this as a 'Mastcam cal target' because this specific region looks like these known prototypes from my training data."
Enhancing Evidence through Diversity
A key innovation in this work is the introduction of a Diversity Loss ($Div$) function. Standard prototypical models often suffer from redundancy, learning multiple prototypes that look nearly identical. The $Div$ term penalizes the model if its prototypes are too similar in the latent space, forcing it to identify a wider variety of visual cues—such as shape, texture, and background context. Interestingly, this diversity didn't just improve interpretability; it also boosted test accuracy by approximately 2%, as the model was forced to consider more robust evidence.
Datasets and Evaluation
The researchers trained and tested their model on the MSL Surface Dataset, which consists of images from the Curiosity rover’s Mastcam and MAHLI (Mars Hand Lens Imager) instruments.
The dataset includes 19 categories (as seen above). To ensure the model's real-world utility, the authors used a "sol-based split," training on data from earlier Martian days (sols) and testing on later ones. This approach accounts for the natural degradation of rover hardware and changes in the Martian environment over time.
Key Results and Insights
- High-Confidence Accuracy: The ResNet18-based Proto-MSLNet achieved an accuracy of 86.39% when the model's confidence was at least 90%.
- The Interpretability Trade-off: There is a slight "interpretability tax"—a roughly 2.5% drop in raw accuracy compared to opaque models like VGG19. However, the authors argue that the resulting transparency and ability to debug the model are far more valuable for scientific applications.
- Negative Evidence: The model also learns "negative weights." For example, as seen in the visualization of the calibration target, it can learn to give negative weight to rocky backgrounds, effectively filtering out noise to focus on relevant man-made features.
Future Directions
The team plans to integrate this interpretable search capability into the live PDS Image Atlas. Future work will focus on "human-in-the-loop" systems, where scientists can provide feedback on prototypes to refine the model's logic. Additionally, the researchers aim to expand the system to include data from other missions, such as the Mars Reconnaissance Orbiter and the Spirit and Opportunity rovers.
Reference: Vasu, B., et al. (2024). Interactive Mars Image Content-Based Search with Interpretable Machine Learning. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI-24).
Summary updated