TL;DR
Deep neural networks (DNNs) are powerful tools for finding gravitational lenses in astronomical surveys, but their "black box" nature makes it hard to understand their limitations. This paper introduces a "sensitivity probe" to quantify how DNN performance changes with input properties, revealing that current lens-finding DNNs are highly sensitive to image color and the simulated point spread function (PSF) used in training, suggesting areas for improvement in training data.
What Problem Does This Paper Address?
The discovery of gravitational lenses, where massive objects bend the light from background sources, is crucial for understanding cosmology and dark matter. While deep neural networks (DNNs) have revolutionized this field, enabling the discovery of thousands of potential lenses, their interpretability remains a significant challenge. It's difficult to understand why a DNN makes a particular classification, what features it's learning, and where its biases lie. This lack of interpretability hinders our ability to fully trust these models, estimate their completeness, and refine their performance. Specifically, astronomers need to understand the "selection function" of these DNNs – which types of lenses are they more or less likely to find, and why? This paper aims to address this by developing and applying a method to probe the sensitivity of DNNs used for gravitational lens finding to various input properties.
Methods
The core of this paper is the development and application of a "sensitivity probe." This method quantifies how the DNN's performance, measured by its "correct class score" (essentially, its confidence in the correct classification), changes when specific input properties are systematically varied.
The probe works in two main ways:
- Binning by Parameter: The test dataset is divided into bins based on a specific physical property (e.g., galaxy magnitude, Einstein radius). The average correct class score is then calculated for each bin. A significant drop in this score as a parameter changes indicates sensitivity.
- Perturbation: Instead of binning, specific input properties are artificially altered (e.g., adding noise, blurring the image, changing colors). The change in the correct class score is then measured as a function of the perturbation's magnitude.
This approach allows researchers to map the DNN's performance landscape and identify which input features are most influential, or where the model breaks down.
Datasets
The study utilizes datasets derived from the Dark Energy Survey (DES).
- Training Data: The DNNs were trained on simulated gravitational lenses. Two training set strategies were employed:
- Network A: Trained on simulated lenses (elliptical galaxies with lensed sources) and non-lenses (elliptical galaxies only).
- Network B: Trained on simulated lenses (elliptical galaxies with lensed sources) and non-lenses (random sources from the field, including various galaxy types and stars).
- Test Data: A comprehensive test set was created, including:
- Simulated lenses.
- Simulated elliptical galaxies.
- Real sources of various types.
- Known lenses and high-quality lens candidates (human-verified).
The paper also mentions a small test set of real strong lenses used for validation.
Key Insights and Ideas
- DNNs are learning "color": The networks are highly sensitive to the color of the simulated lenses. This suggests they've learned to associate typical lens colors (e.g., red elliptical lens with a blue star-forming source) and may struggle with atypical color combinations.
- PSF sensitivity is a weakness: The networks are overly sensitive to the specific Point Spread Function (PSF) used during training. Performance degrades significantly if the PSF is better or worse than the training PSF, indicating a potential bias and a need for more diverse PSF simulations in training.
- Einstein radius is not a key feature: The DNNs show remarkable insensitivity to the Einstein radius of simulated lenses, suggesting this geometric parameter is not a primary driver of their classification decisions.
- Occlusion reveals learned features: Masking out specific regions of an image (occlusion) shows that the networks rely heavily on information within a radius of approximately 1.3 arcseconds, which corresponds to the typical Einstein radius of detectable lenses.
- Network B is more robust to noise: Network B, trained on a more diverse set of non-lenses, exhibits a different strategy when faced with noise compared to Network A. While Network A becomes uncertain, Network B becomes more confident that the object is not a lens, suggesting fewer false positives in low signal-to-noise regimes.
Key Results
- Color Jitter: Deviating from the training set's color distribution by more than 20% significantly degrades network accuracy.
- PSF: Performance degrades smoothly as the PSF widens. Network B is more affected by PSF changes than Network A. Notably, performance also degrades if the PSF is better than the training PSF, suggesting the networks are "smart" enough to avoid unphysically sharp features but are also overly sensitive to the training PSF.
- Magnitudes: Network A's accuracy degrades significantly for source g-band magnitudes fainter than 21.5. Network B shows a similar trend but is slightly more robust. For lens r-band magnitude, Network A degrades for bright lenses (r < 19), while Network B degrades for fainter lenses (r > 21).
- Noise and Blur: As expected, adding noise or blurring the images smoothly decreases network performance, as information is removed.
- Occlusion: For simulated lenses, accuracy drops to ~75% when information within a radius of 0.5-1.7 arcseconds is removed, confirming the importance of this region. Non-lens scores are unaffected.
Limitations and Future Work
- Limitations of the Sensitivity Probe: The method requires a priori knowledge of the properties to test and doesn't automatically discover hidden biases. It also primarily analyzes correlations between biases and observational properties, rather than complex multi-dimensional relationships. Errors in catalog properties can also affect results.
- Future Work:
- Improved Training Data: The authors strongly suggest improving training sets by:
- Including a wider range of galaxy colors to avoid over-sensitivity.
- Matching the PSF more closely to the target imaging and including a broader distribution of PSFs.
- Quantifying Assumptions: Future work should focus on quantifying the significance of assumptions made in simulations, such as the Sersic index of simulated sources, the presence of multiple lensed sources, and clumpiness.
- More Complex Models: Testing more complex mass distributions in the lens plane could be beneficial.
- Broader Application: The sensitivity probe method could be valuable for other DNN applications in astronomy and beyond, helping to inform experimental design and identify training set weaknesses.
- Bayesian Neural Networks: Combining the sensitivity probe with techniques for estimating uncertainties in ANN outputs could further enhance interpretability.
Summary updated