Radio Galaxy Zoo: Towards building the first multi-purpose foundation model for radio astronomy with self-supervised learning

Inigo Val Slijepcevic, Anna M. M. Scaife, Mike Walmsley, Micah Bowles, O. Ivy Wong, Stanislav S. Shabala, S. V. White

TL;DR

Researchers have developed the first multi-purpose "foundation model" for radio astronomy using self-supervised learning (SSL). By pre-training on over 100,000 unlabeled galaxy images, the model achieves state-of-the-art classification accuracy with minimal human labeling and generalizes across different telescope surveys, effectively halving the error rates of traditional supervised methods.

What Problem Does This Paper Address?

Radio astronomy is entering a "Big Data" era with next-generation telescopes like the Square Kilometre Array (SKA) set to produce unprecedented volumes of data. Traditionally, astronomers have relied on supervised Deep Learning (CNNs) to classify galaxy morphologies—specifically the Fanaroff-Riley (FR) types.

However, supervised learning has three major "bottlenecks":

  1. Label Scarcity: Expert labeling is expensive and slow; we have millions of images but only thousands of labels.
  2. Selection Bias: Labeled datasets often favor bright, nearby, or "obvious" sources, causing models to struggle with the messy reality of new surveys.
  3. Rigidity: A model trained to distinguish "Type A" from "Type B" must be entirely retrained if a scientist wants to look for "Type C" or perform a different task like similarity searching.

The authors aim to build a foundation model: a single, robust representation of radio images that can be "recycled" for many different scientific tasks with little to no extra training.

Methods

The team utilized Self-Supervised Learning (SSL), specifically an algorithm called Bootstrap Your Own Latent (BYOL).

Unlike standard training where a model is told "this is an FRI galaxy," BYOL uses instance differentiation. The model is shown two different "views" of the same unlabeled image (created via augmentations like rotation, cropping, and blurring) and tasked with recognizing that they represent the same object. To do this, it must learn the underlying structural features of galaxies—essentially teaching itself what a galaxy looks like without human intervention.

The architecture uses a standard ResNet-18 backbone. Once pre-trained on a massive unlabeled pool, the model can be "fine-tuned" on a tiny set of labeled data or used directly for tasks like similarity searches.

Datasets

  • RGZ DR1 (Radio Galaxy Zoo Data Release 1): A massive, uncurated set of 107,893 images from the VLA FIRST survey used for the initial self-supervised pre-training.
  • MiraBest: A high-quality labeled dataset (1,256 sources) used to evaluate classification performance.
  • MIGHTEE: Data from the newer MeerKAT telescope. This served as an "out-of-distribution" test to see if a model trained on one telescope's data could work on another's without extra tuning.

Key Insights and Ideas

  • Emergent Physics: Even though the model was never told about the physical properties of galaxies, UMAP visualizations show that its internal representation space naturally organized itself by angular extent (source size). The model "discovered" a physical attribute just by looking at the data.
  • The "Foundation" Advantage: The model isn't just better at classification; it's a multi-tool. The authors demonstrated a similarity search where, given one rare "hybrid" galaxy, the model could instantly find others in the 100k+ image pool without needing any labels.
  • Compute vs. Accuracy Trade-off: For researchers without massive GPU clusters, the authors found that fine-tuning just the final layer of the pre-trained model yields results comparable to full supervised training but at a fraction of the computational cost and time.
  • Augmentation is Key: In radio astronomy, images are greyscale and sparse. The authors found that random resized cropping and rotation are the most critical augmentations for helping the model learn robust features.

Key Results

  • Superior Accuracy: The SSL-pre-trained model reduced the test set error on the MiraBest dataset by up to half compared to a purely supervised baseline.
  • Extreme Label Efficiency: The model achieved a remarkably low 7.79% error rate using only 145 labels. For comparison, a standard supervised model usually requires thousands of labels to reach similar stability.
  • Cross-Survey Generalization: When tested on the MIGHTEE survey (different telescope, different noise characteristics), the foundation model improved accuracy by ~8% over the baseline without any hyper-parameter tuning.
  • Scientific Discovery: The model successfully identified rare hybrid morphology sources in the unlabeled RGZ DR1 dataset that had previously gone unnoticed.

Limitations and Future Work

The authors acknowledge that while the model is robust, the current augmentation pipeline is somewhat limited by the nature of radio images (e.g., no color channels). They also note that the "unlabeled" pool still has some selection bias because it consists of pre-cut "stamps" rather than raw, full-sky maps.

Future Work: The team aims to integrate multi-modal data—combining radio images with optical data or spectral information—to build an even more comprehensive foundation model for extragalactic astronomy. They have also released an interactive web app for astronomers to explore the model's latent space.

Summary updated

Abstract

In this work, we apply self-supervised learning with instance differentiation to learn a robust, multi-purpose representation for image analysis of resolved extragalactic continuum images. We train a multi-use model which compresses our unlabelled data into a structured, low dimensional representation which can be used for a variety of downstream tasks (e.g. classification, similarity search). We exceed baseline supervised Fanaroff-Riley classification performance by a statistically significant margin, with our model reducing the test set error by up to half. Our model is also able to maintain high classification accuracy with very few labels, with only 7.79% error when only using 145 labels. We further demonstrate that by using our foundation model, users can efficiently trade off compute, human labelling cost and test set accuracy according to their respective budgets, allowing for efficient classification in a wide variety of scenarios. We highlight the generalizability of our model by showing that it enables accurate classification in a label scarce regime with data from the new MIGHTEE survey without any hyper-parameter tuning, where it improves upon the baseline by ~8%. Visualizations of our labelled and un-labelled data show that our model's representation space is structured with respect to physical properties of the sources, such as angular source extent. We show that the learned representation is scientifically useful even if no labels are available by performing a similarity search, finding hybrid sources in the RGZ DR1 data-set without any labels. We show that good augmentation design and hyper-parameter choice can help achieve peak performance, while emphasising that optimal hyper-parameters are not required to obtain benefits from self-supervised pre-training.
BibTeX
@inproceedings{slijepcevic2023radio,
  author = {Inigo Val Slijepcevic and Anna M. M. Scaife and Mike Walmsley and Micah Bowles and O. Ivy Wong and Stanislav S. Shabala and S. V. White},
  title = {{Radio Galaxy Zoo: Towards building the first multi-purpose foundation model for radio astronomy with self-supervised learning}},
  booktitle = {RAS},
  year = {2023},
  doi = {10.1093/rasti/rzad055},
}
Figure