TL;DR
Deep learning models, while powerful for analyzing astronomical data, are surprisingly brittle to common image imperfections like noise and compression. This paper demonstrates that domain adaptation techniques can significantly improve the robustness of these models, making them more reliable for scientific discovery.
What Problem Does This Paper Address?
Deep learning models excel at handling the vast and complex datasets generated by modern scientific instruments, such as the Legacy Survey of Space and Time (LSST) in astronomy. However, these models can be "brittle," meaning their performance degrades significantly when faced with slight, often unintentional, changes to the input data. These changes can arise from various sources, including image compression, blurring, or even subtle variations in observational noise due to differing exposure times. Such imperfections are common in real-world scientific data. The paper aims to understand how these "adversarial perturbations," which mimic natural data imperfections, affect deep learning models used for galaxy morphology classification and to explore methods for making these models more robust.
Methods
The researchers trained a standard ResNet18 convolutional neural network, a common choice for image classification tasks, to distinguish between spiral, elliptical, and merging galaxies. They then subjected this trained model to two types of "attacks":
- Observational Noise: They simulated data with varying levels of noise, mimicking either one year (Y1) or ten years (Y10) of LSST observations. This allowed them to study the impact of signal-to-noise ratio on model performance.
- One-Pixel Attack: This is a type of "black-box" attack where only a single pixel in an image is modified. This serves as a proxy for potential errors introduced by data compression or telescope hardware malfunctions. The specific pixel to be modified is found using a population-based optimization algorithm called differential evolution.
To improve model robustness, they employed Domain Adaptation techniques, specifically Maximum Mean Discrepancy (MMD). MMD is a method that aligns the latent space representations of data from different domains. In this case, they treated the cleaner Y10 data as the "labeled domain" and the noisier Y1 data as the "unlabeled domain." By minimizing the MMD between these domains during training, the network learns to extract features that are invariant to the noise, thus becoming more robust.
Datasets
The study utilized mock galaxy images generated from the IllustrisTNG100 simulation. Specifically, they extracted three-filter (g, r, i) images from snapshots at redshifts z = 0.05 and z = 0. These images were then processed using the GalSim package to simulate LSST observations, incorporating both observational noise (Poisson distribution) and point spread function (PSF) blurring. Galaxy morphology labels were derived from the IllustrisTNG morphology catalogs and further refined using G-M20 statistics. The final dataset comprised 14,312 spiral, 8,151 elliptical, and 12,710 merging galaxy images, split into training (70%), validation (10%), and test (20%) sets.
Key Insights and Ideas
- Natural Imperfections are Adversarial: The paper highlights that common data imperfections like noise and single-pixel errors, which are often unavoidable in scientific data collection and processing, can act as adversarial attacks, significantly degrading model performance.
- Domain Adaptation as a Robustness Tool: Domain adaptation, particularly MMD, proves to be a powerful technique for enhancing model robustness. By learning domain-invariant features, the model becomes less sensitive to variations in data quality.
- Visualizing Latent Space is Crucial: Techniques like "church window plots" and isomaps are invaluable for understanding how perturbations affect the model's internal representations and why misclassifications occur. They provide a visual intuition for the model's decision-making process.
- One-Pixel Attacks are Surprisingly Effective: Even a single pixel modification can be enough to fool a well-trained model, underscoring the need for robust architectures and training strategies.
Key Results
- Significant Performance Drop Without Adaptation: Without domain adaptation, the accuracy of the ResNet18 model on noisy Y1 data dropped from 72% to 43%.
- Domain Adaptation Boosts Robustness: With domain adaptation, the accuracy on Y10 data increased to 82%, and crucially, the accuracy on noisy Y1 data rose to 66%. This represents a substantial improvement in robustness.
- Increased Perturbation Distance: For images successfully flipped by a one-pixel attack, domain adaptation required the perturbation to be approximately 2.3 times larger in the latent space, indicating a more resilient model.
- Improved Classification Metrics: Table 1 shows that domain adaptation consistently improved accuracy, precision, recall, and F1 scores for both Y10 and Y1 datasets.
Limitations and Future Work
The authors acknowledge that while domain adaptation shows promise, it's not a complete solution for all adversarial attacks. They note that even with domain adaptation, the model could still be fooled by the one-pixel attack, although it required more iterations to find the effective pixel, and the attacked image was further from the original in the latent space.
Future work could involve exploring other domain adaptation techniques, as well as architectural improvements and data augmentation strategies, to further enhance model robustness. The paper suggests that these methods can be combined to build even more trustworthy and stable models for scientific applications.
Summary updated