Radio galaxy zoo: towards building the first multipurpose foundation model for radio astronomy with self-supervised learning

Inigo Val Slijepcevic, Anna M. M. Scaife, Mike Walmsley, Micah Bowles, O. Ivy Wong, Stanislav S. Shabala, S. V. White

TL;DR

This paper introduces a self-supervised learning (SSL) "foundation model" for radio galaxy image analysis, significantly improving classification accuracy, especially with limited labels. By leveraging vast amounts of unlabeled radio survey data, this model offers a more efficient and robust approach to understanding radio galaxies, enabling new scientific applications like similarity searches.

What Problem Does This Paper Address?

Traditional radio galaxy classification relies heavily on supervised learning, which requires large, meticulously labeled datasets. This approach faces several challenges:

  • Selection Bias: Labeled datasets are often curated, introducing biases that limit generalization to unseen data.
  • Costly Labeling: Acquiring labels for astronomical data is expensive and time-consuming, especially as new, high-sensitivity telescopes generate more data.
  • Implicit Bias: Supervised models are inherently biased towards the predefined classification schemes they are trained on, which may not align with emerging scientific needs or new data characteristics.
  • Computational Inefficiency: Retraining models from scratch for new tasks or datasets is computationally intensive.

The authors aim to overcome these limitations by developing a "foundation model" using self-supervised learning, which can learn robust representations from unlabeled data and be efficiently adapted for various downstream tasks.

Methods

The core of this work is the application of Bootstrap Your Own Latent (BYOL), a self-supervised learning algorithm. BYOL trains an encoder to learn a low-dimensional representation of input data by minimizing the distance between different augmented views of the same image. Unlike some other SSL methods, BYOL does not require "negative pairs" (augmentations from different images), which reduces computational cost and is beneficial for data with less distinct class boundaries, like radio galaxy images.

Key methodological aspects include:

  • Custom Augmentation Pipeline: The standard ImageNet augmentation pipeline was adapted for radio galaxy images, considering their unique characteristics (e.g., greyscale, fine-grained details, potential for information destruction with aggressive augmentation).
  • Foundation Model Concept: The goal is to train a single, versatile model on a large unlabeled dataset that can then be fine-tuned for various specific tasks with minimal labeled data.
  • Linear Evaluation: To assess the quality of the learned representations, a linear classifier is trained on top of the frozen encoder. This method tests how well the representation space separates different classes without further training the encoder.
  • Fine-tuning: For maximum performance on downstream tasks, the pre-trained model's layers are further trained (fine-tuned) using labeled data.

Datasets

  • RGZ DR1 (Radio Galaxy Zoo Data Release 1): A large, uncurated dataset of radio galaxy images used for self-supervised pre-training. This dataset contains over 107,000 images.
  • MiraBest: A curated dataset of radio galaxy images with Fanaroff-Riley (FRI/FRII) classifications. This dataset was used for supervised training, fine-tuning, and evaluation. It contains 1,256 "Confident" samples and additional "Uncertain" samples.
  • MIGHTEE Survey: A smaller, expert-labeled dataset from a different telescope (MeerKAT) used to test the model's cross-survey generalization capabilities. This dataset consists of 72 FRII and 45 FRI sources.

Key Insights and Ideas

  • SSL for Radio Astronomy is Viable: The paper demonstrates that self-supervised learning, specifically BYOL, can effectively learn meaningful representations from unlabeled radio galaxy images, outperforming traditional supervised methods in many scenarios.
  • Unlabeled Data Bridges the Gap: By training on a vast amount of unlabeled data, the model learns a representation space that is more representative of the full data manifold, leading to better generalization and reduced reliance on labeled data.
  • Efficiency and Flexibility: The foundation model approach allows users to trade off computational resources, labeling effort, and desired accuracy. This is crucial for making advanced analysis accessible.
  • Representation Space Structure: Visualizations show that the learned representation space is structured according to physical properties of the sources (e.g., angular extent), even though this information was not explicitly provided during training.
  • Similarity Search Capability: The learned representations are useful for scientific tasks like similarity search, enabling the discovery of rare or unusual objects (e.g., hybrid sources) without explicit labels.

Key Results

  • Improved Classification Accuracy: The self-supervised foundation model, when fine-tuned, significantly outperforms the supervised baseline, reducing test set error by up to half on the MiraBest dataset.
  • Label Efficiency: With only 145 labels, the model achieves a classification error of 7.79%, a substantial improvement over the supervised baseline. With 291 labels, the pre-trained model achieves the same accuracy as the supervised model trained on all 729 available labels.
  • Cross-Survey Generalization: The model shows an ~8% improvement in accuracy on the MIGHTEE dataset compared to a supervised baseline trained from scratch, demonstrating its ability to generalize across different surveys and telescopes.
  • Similarity Search Success: The model successfully identified hybrid radio galaxy sources in the RGZ DR1 dataset using only a single example, showcasing its utility for scientific discovery.
  • Robustness to Augmentation: While optimal augmentation design can improve performance, the model's fine-tuned performance is relatively robust to the omission of some augmentations, making it easier to apply to new datasets.

Limitations and Future Work

  • Augmentation Design: While the model is robust, optimal augmentation design is still important, especially for linear evaluation or when fine-tuning only a few layers.
  • Data Curation: While the model handles uncurated data well, some pre-processing (e.g., angular extent cutoffs) was still applied to improve performance.
  • Future Work: The authors suggest integrating data from a wider range of sources and modalities (e.g., spectral information) to create even more comprehensive foundation models. They also envision applying this approach to a greater variety of tasks and learning relationships between different data types. The development of an interactive app for similarity search is also highlighted as a way to make the model accessible.

Summary updated

Abstract

Abstract In this work, we apply self-supervised learning with instance differentiation to learn a robust, multipurpose representation for image analysis of resolved extragalactic continuum images. We train a multi-use model which compresses our unlabelled data into a structured, low dimensional representation which can be used for a variety of downstream tasks (e.g. classification, similarity search). We exceed baseline supervised Fanaroff–Riley classification performance by a statistically significant margin, with our model reducing the test set error by up to half. Our model is also able to maintain high classification accuracy with very few labels, with only $7.79{{\ \rm per\ cent}}$ error when only using 145 labels. We further demonstrate that by using our foundation model, users can efficiently trade off compute, human labelling cost and test set accuracy according to their respective budgets, allowing for efficient classification in a wide variety of scenarios. We highlight the generalizability of our model by showing that it enables accurate classification in a label scarce regime with data from the new MIGHTEE survey without any hyperparameter tuning, where it improves upon the baseline by $\sim 8{{\ \rm per\ cent}}$. Visualizations of our labelled and un-labelled data show that our model’s representation space is structured with respect to physical properties of the sources, such as angular source extent. We show that the learned representation is scientifically useful even if no labels are available by performing a similarity search, finding hybrid sources in the RGZ DR1 data set without any labels. We show that good augmentation design and hyperparameter choice can help achieve peak performance, while emphasizing that optimal hyperparameters are not required to obtain benefits from self-supervised pre-training.
BibTeX
@inproceedings{slijepcevic2023radio,
  author = {Inigo Val Slijepcevic and Anna M. M. Scaife and Mike Walmsley and Micah Bowles and O. Ivy Wong and Stanislav S. Shabala and S. V. White},
  title = {{Radio galaxy zoo: towards building the first multipurpose foundation model for radio astronomy with self-supervised learning}},
  booktitle = {RAS},
  year = {2023},
  doi = {10.1093/rasti/rzad055},
}
Figure