Galaxy Morphology Classification with Counterfactual Explanation

Zhuo Cao, Lena Krieger, Hanno Scharr, Ira AssentarXiv 2025 · DOI · arXiv

Introduction

Understanding galaxy morphology—the study of shape and structure—is fundamental to astrophysics, offering insights into evolutionary history and cosmic dynamics. With modern surveys producing millions of images, Convolutional Neural Networks (CNNs) have become the standard for automated classification. However, the "black-box" nature of these models often obscures the physical features driving their decisions, limiting scientific trust. This work addresses this gap by generating visual counterfactual explanations (CEs): synthetic images that illustrate the minimal, semantically meaningful changes needed to flip a model's prediction from one morphological class to another.

Methodology

The proposed architecture replaces the standard classification head with an Invertible Neural Network (INN) to enable bidirectional mapping between image features and class representations.

Overview of the training and explaining phases of the counterfactual generation framework.

The process involves three core components:

  1. Latent Disentanglement: An encoder maps the input image into a latent space split into a class-dependent vector ($z_1$) and a class-independent vector ($z_2$). This ensures that background details are preserved during counterfactual generation.
  2. Invertible Normalizing Flow: The $z_1$ vector is processed through an invertible flow ($F$) that maps it to a Gaussian Mixture Model (GMM). Classification is performed based on the proximity to cluster centers in this hidden space.
  3. Counterfactual Synthesis: To generate an explanation, the model shifts a galaxy's hidden representation across a decision boundary into a target class's cluster. The inverse flow ($F^{-1}$) then produces a modified $z_1^{cf}$, which is combined with the original $z_2$ and passed through a decoder to reconstruct the counterfactual image.

The model is trained using a composite loss function including perceptual reconstruction loss, Maximum Mean Discrepancy (MMD) for latent continuity, and an Information Bottleneck loss to ensure $z_1$ contains only essential classification features.

Experimental Results

The model was evaluated on the Galaxy10 DECaLS dataset, comprising 17,736 images across 10 morphological classes.

Classification Performance

The framework achieves an overall accuracy of approximately 80%. While most classes perform well (70-90%), the model struggles with "Disturbed" galaxies (41% accuracy), likely due to their chaotic and diffuse nature which prevents clean clustering in the latent space.

Counterfactual Realism and Interpretability

The generated counterfactuals are highly realistic, maintaining a Structural Similarity Index Measure (SSIM) of 0.96 relative to the original images.

Grid of counterfactual transformations showing how the model modifies specific features to change class predictions.

As shown in the figure above, the model successfully isolates and modifies class-defining features. For instance, converting a "round smooth" galaxy to a "cigar-shaped" one involves a clear elongation of the central light distribution while keeping the background noise identical. This demonstrates effective feature disentanglement.

Discussion and Limitations

A key insight is the model's ability to visualize its internal decision boundaries. However, the use of an Information Bottleneck introduces a trade-off: the compression required for robust classification sometimes discards fine-grained details like delicate spiral arms. Consequently, counterfactuals between similar spiral sub-types can appear visually indistinguishable.

Future work aims to refine the architecture to better capture high-frequency morphological structures. Additionally, the sensitivity of the latent distributions to mislabeled data suggests a potential secondary use-case: automatically identifying and flagging noisy labels in large-scale astronomical datasets.

Conclusion

By combining invertible flows with latent space splitting, this framework provides a powerful tool for interpretable galaxy classification. It moves beyond simple accuracy metrics, offering astrophysicists visual evidence of the morphological features that drive machine learning predictions.

Summary updated

Abstract

Galaxy morphologies play an essential role in the study of the evolution of galaxies. The determination of morphologies is laborious for a large amount of data giving rise to machine learning-based approaches. Unfortunately, most of these approaches offer no insight into how the model works and make the results difficult to understand and explain. We here propose to extend a classical encoder-decoder architecture with invertible flow, allowing us to not only obtain a good predictive performance but also provide additional information about the decision process with counterfactual explanations.
BibTeX
@misc{cao2025galaxy,
  author = {Zhuo Cao and Lena Krieger and Hanno Scharr and Ira Assent},
  title = {{Galaxy Morphology Classification with Counterfactual Explanation}},
  year = {2025},
  eprint = {2510.14655},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.14655},
}
Figure