Science Pipelines for the Square Kilometre Array

J. S. Farnes, Ben Mort, Fred Dulwich, S. Salvini, Wesley Armour

TL;DR

This paper details the development of an SKA-like science pipeline designed to handle the immense data volumes from future radio telescopes. By leveraging the LOFAR telescope and the Dask parallel computing library, the team has created a scalable system for spectral imaging and Faraday tomography, crucial for studying cosmic magnetic fields. This work is vital for enabling cutting-edge astronomical discoveries with the SKA and advancing our understanding of the Universe's large-scale structure.

What Problem Does This Paper Address?

The Square Kilometre Array (SKA) promises to be the most powerful radio telescope ever built, generating an unprecedented amount of data – an estimated 5 zettabytes per year in its first phase. This sheer volume of data presents a significant challenge for processing and analysis. Traditional methods are insufficient, necessitating the development of highly efficient and scalable "science pipelines" to extract meaningful scientific information. Specifically, this paper focuses on the challenge of processing polarization data to study the faint magnetic fields within the "cosmic web" – the large-scale structure of filaments and voids in the Universe. Measuring these fields requires sophisticated techniques like Faraday Tomography, which are computationally intensive and demand robust data processing capabilities.

Methods

The core of this work is the development of an SKA-like science pipeline. The authors are building this pipeline using Python and the Dask distributed computing library. Dask is chosen for its ability to scale parallel computations across clusters, even those with thousands of cores, and its flexibility in implementing task-based parallelism with minimal code changes. The pipeline is designed to handle the complex requirements of full-Stokes polarization imaging and Faraday Tomography. This involves processing every spectral channel of the observed data, which is a highly parallelizable task. The pipeline incorporates algorithms for ionospheric Faraday rotation calibration, complex CLEAN deconvolution for polarimetric data, Faraday Moment image generation for source finding, and RM-CLEAN for Faraday Tomography. The authors are also exploring the potential for deploying these algorithms on Graphics Processing Units (GPUs) for further acceleration.

Datasets

The primary dataset used for testing and validating the developed pipeline is data from the LOFAR Multifrequency Snapshot Sky Survey (MSSS) and its polarized counterpart, the MSSS All-Sky Polarization Survey (MAPS). This survey provides approximately 50 terabytes of archived visibility data, which the authors aim to image at 45 arcsec angular resolution. While not a newly introduced benchmark in the traditional sense, the MSSS/MAPS dataset serves as a realistic testbed for SKA-like data volumes and processing requirements. The authors also mention that the developed infrastructure could be applied to data from other SKA pathfinder and precursor telescopes, such as ASKAP and the Karl G. Jansky Very Large Array (VLA).

Key Insights and Ideas

  • The SKA is a Big Data challenge, not just a big telescope: The sheer scale of data generation from the SKA necessitates a paradigm shift in how astronomical data is processed. This paper highlights that developing efficient pipelines is as crucial as building the telescope itself.
  • Faraday Tomography is a computationally demanding but scientifically rich technique: The study of cosmic magnetic fields, particularly on large scales, relies heavily on Faraday Tomography. This technique, while powerful, presents significant computational hurdles that need to be overcome for SKA-scale datasets.
  • Dask is a promising tool for SKA-scale data processing: The paper demonstrates that Dask can effectively parallelize complex astronomical processing tasks, offering a scalable and flexible solution for handling the massive data streams expected from the SKA.
  • Prototyping with existing telescopes is essential: Using LOFAR data as a proxy for SKA data allows for the development and validation of pipelines before the SKA is fully operational, ensuring readiness for scientific discovery.
  • The "cosmic web" holds secrets about magnetic fields: The research aims to probe the faint magnetic fields within the large-scale structure of the Universe, which could reveal fundamental physics and the origins of magnetism.

Key Results

The paper reports on the initial testing of a parallelized version of their SKA-like science pipeline. A significant speedup was observed: a serialised code that took 11 hours to process a single sky pointing was reduced to just 8 minutes using the parallelized Dask pipeline. Furthermore, source parameter measurements (sky coordinates and flux density) for a detected pulsar were found to be consistent with those obtained from other radio astronomy packages, validating the technical accuracy and scientific reliability of their pipeline.

Limitations and Future Work

The authors acknowledge that the current work is an ongoing development effort. They aim to have a fully working and deployable spectral imaging and Faraday Tomography pipeline by the end of 2018. Future work will involve further testing the scalability and implementation of these SKA-like science pipelines, optimizing the Faraday Tomography algorithm for SKA sensitivities, and potentially deploying these algorithms on GPUs. The ultimate goal is to enable tight constraints on the magnetised large-scale structure of the Universe and contribute to cutting-edge scientific discoveries.

Summary updated

Abstract

The Square Kilometre Array (SKA) will be both the largest radio telescope ever constructed and the largest Big Data project in the known Universe. The first phase of the project will generate on the order of five zettabytes of data per year. A critical task for the SKA will be its ability to process data for science, which will need to be conducted by science pipelines. Together with polarization data from the LOFAR Multifrequency Snapshot Sky Survey (MSSS), we have been developing a realistic SKA-like science pipeline that can handle the large data volumes generated by LOFAR at 150 MHz. The pipeline uses task-based parallelism to image, detect sources and perform Faraday tomography across the entire LOFAR sky. The project thereby provides a unique opportunity to contribute to the technological development of the SKA telescope, while simultaneously enabling cutting-edge scientific results. In this paper, we provide an update on current efforts to develop a science pipeline that can enable tight constraints on the magnetised large-scale structure of the Universe.
BibTeX
@inproceedings{farnes2018science,
  author = {J. S. Farnes and Ben Mort and Fred Dulwich and S. Salvini and Wesley Armour},
  title = {{Science Pipelines for the Square Kilometre Array}},
  booktitle = {Galaxies},
  year = {2018},
  doi = {10.3390/galaxies6040120},
}
Figure