Learning useful representations for radio astronomy "in the wild" with contrastive learning

Inigo Val Slijepcevic, Anna M. M. Scaife, Mike Walmsley, Micah BowlesarXiv 2022 · DOI · arXiv

TL;DR

Researchers have successfully applied self-supervised contrastive learning to "in the wild" radio astronomy data, creating a model that learns to categorize galaxies without needing a single human-provided label. By carefully filtering out small, unresolved sources, the team achieved classification performance that rivals supervised models while enabling powerful new capabilities like similarity searches for rare cosmic objects.

What Problem Does This Paper Address?

Radio astronomy is entering the era of "Big Data." Next-generation telescopes like the Square Kilometre Array (SKA) will produce exabytes of data, revealing millions of new radio galaxies. Traditionally, astronomers classify these galaxies into categories like Fanaroff-Riley I and II (FRI/FRII) based on their shapes.

However, supervised deep learning—the current gold standard—requires massive amounts of labeled data. Labeling is expensive, slow, and prone to human bias. While semi-supervised learning (using a mix of labeled and unlabeled data) seems like a solution, it often breaks down when the unlabeled "wild" data has a different distribution than the labeled training set—a phenomenon known as dataset shift. This paper explores whether self-supervised representation learning can bridge this gap by learning the underlying structure of the data itself, without needing labels upfront.

Methods

The authors utilized a contrastive learning framework called BYOL (Bootstrap Your Own Latent). Unlike other popular methods (like SimCLR), BYOL does not require "negative pairs" (explicitly telling the model which images are different). Instead, it uses two neural networks—an "online" and a "target" network—that learn from different augmented views of the same image.

Key technical choices included:

  • Backbone: A ResNet-18 architecture.
  • Augmentation Tweaks: The authors found that standard computer vision augmentations (like heavy blurring) destroyed the fine-grained morphological details of radio galaxies. They reduced the blurring kernel size and probability to preserve scientific information.
  • Data Curation: The most critical step was implementing an arcsecond cut threshold. They removed sources with low angular extent (small, unresolved dots) because these sources lack the morphological features the model needs to learn useful representations.

Datasets

The study utilized two primary datasets:

  1. MiraBest: A high-quality, human-labeled dataset of 1,256 radio galaxies used for evaluation and as a supervised baseline.
  2. Radio Galaxy Zoo Data Release 1 (RGZ DR1): A much larger, "wild" dataset containing ~100,000 unlabeled sources from the FIRST survey. This served as the primary training set for the self-supervised model.

Key Insights and Ideas

  • Curation is King: The most striking insight is that "more data" isn't always better. Including small, unresolved sources in the training set caused the model's performance to plateau. By removing sources smaller than ~25 arcseconds, the model was forced to focus on complex shapes, leading to a "qualitatively different training dynamic."
  • Closing the Gap: Even though the model never saw a label during training, a simple k-nearest neighbor (kNN) classifier applied to its learned features could separate FRI and FRII galaxies with high accuracy.
  • Beyond Classification: Because the model learns a general "representation space," it can be used for tasks other than classification. The authors demonstrated this via similarity searches, where the model could find rare "hybrid" galaxies or even identify duplicate entries in the catalog that had escaped previous automated filters.

Key Results

  • Accuracy: The self-supervised model achieved a classification accuracy of 85.25% on the MiraBest test set. While lower than the supervised baseline (92.42%), it is remarkably close considering the self-supervised model used zero labels and was trained on a completely different dataset (RGZ DR1).
  • The Threshold Effect: Performance was highly sensitive to the angular extent cut. At low thresholds (including small sources), validation accuracy remained flat. As the threshold increased toward 25–30 arcseconds, accuracy improved by over 10%.
  • Discovery: The model successfully retrieved semantically similar "hybrid" sources (objects sharing both FRI and FRII traits) and identified duplicates in the RGZ DR1 catalog that were slightly offset or cropped differently.

Limitations and Future Work

The authors acknowledge that their current data curation (the arcsecond cut) relies on domain-specific knowledge. A major goal for future work is to find domain-agnostic ways to automatically filter or weight data so that models can learn effectively from even noisier "wild" datasets.

Additionally, while this work focuses on the FRI/II distinction, the ultimate goal is to develop "foundation models" for astronomy—large-scale models that can be fine-tuned for a variety of downstream tasks, from identifying black hole jets to discovering entirely new classes of astronomical phenomena.

Summary updated

Abstract

Unknown class distributions in unlabelled astrophysical training data have previously been shown to detrimentally affect model performance due to dataset shift between training and validation sets. For radio galaxy classification, we demonstrate in this work that removing low angular extent sources from the unlabelled data before training produces qualitatively different training dynamics for a contrastive model. By applying the model on an unlabelled data-set with unknown class balance and sub-population distribution to generate a representation space of radio galaxies, we show that with an appropriate cut threshold we can find a representation with FRI/FRII class separation approaching that of a supervised baseline explicitly trained to separate radio galaxies into these two classes. Furthermore we show that an excessively conservative cut threshold blocks any increase in validation accuracy. We then use the learned representation for the downstream task of performing a similarity search on rare hybrid sources, finding that the contrastive model can reliably return semantically similar samples, with the added bonus of finding duplicates which remain after pre-processing.
BibTeX
@misc{slijepcevic2022learning,
  author = {Inigo Val Slijepcevic and Anna M. M. Scaife and Mike Walmsley and Micah Bowles},
  title = {{Learning useful representations for radio astronomy "in the wild" with contrastive learning}},
  year = {2022},
  eprint = {2207.08666},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.08666},
}

Loading related content...

Figure