TL;DR
This paper details the development of an SKA-like science pipeline designed to handle the immense data volumes from future radio telescopes. By leveraging the LOFAR telescope and the Dask parallel computing library, the team has created a scalable system for spectral imaging and Faraday tomography, crucial for studying cosmic magnetic fields. This work is vital for enabling cutting-edge astronomical discoveries with the SKA and advancing our understanding of the Universe's large-scale structure.
What Problem Does This Paper Address?
The Square Kilometre Array (SKA) promises to be the most powerful radio telescope ever built, generating an unprecedented amount of data – an estimated 5 zettabytes per year in its first phase. This sheer volume of data presents a significant challenge for processing and analysis. Traditional methods are insufficient, necessitating the development of highly efficient and scalable "science pipelines" to extract meaningful scientific information. Specifically, this paper focuses on the challenge of processing polarization data to study the faint magnetic fields within the "cosmic web" – the large-scale structure of filaments and voids in the Universe. Measuring these fields requires sophisticated techniques like Faraday Tomography, which are computationally intensive and demand robust data processing capabilities.
Methods
The core of this work is the development of an SKA-like science pipeline. The authors are building this pipeline using Python and the Dask distributed computing library. Dask is chosen for its ability to scale parallel computations across clusters, even those with thousands of cores, and its flexibility in implementing task-based parallelism with minimal code changes. The pipeline is designed to handle the complex requirements of full-Stokes polarization imaging and Faraday Tomography. This involves processing every spectral channel of the observed data, which is a highly parallelizable task. The pipeline incorporates algorithms for ionospheric Faraday rotation calibration, complex CLEAN deconvolution for polarimetric data, Faraday Moment image generation for source finding, and RM-CLEAN for Faraday Tomography. The authors are also exploring the potential for deploying these algorithms on Graphics Processing Units (GPUs) for further acceleration.
Datasets
The primary dataset used for testing and validating the developed pipeline is data from the LOFAR Multifrequency Snapshot Sky Survey (MSSS) and its polarized counterpart, the MSSS All-Sky Polarization Survey (MAPS). This survey provides approximately 50 terabytes of archived visibility data, which the authors aim to image at 45 arcsec angular resolution. While not a newly introduced benchmark in the traditional sense, the MSSS/MAPS dataset serves as a realistic testbed for SKA-like data volumes and processing requirements. The authors also mention that the developed infrastructure could be applied to data from other SKA pathfinder and precursor telescopes, such as ASKAP and the Karl G. Jansky Very Large Array (VLA).
Key Insights and Ideas
- The SKA is a Big Data challenge, not just a big telescope: The sheer scale of data generation from the SKA necessitates a paradigm shift in how astronomical data is processed. This paper highlights that developing efficient pipelines is as crucial as building the telescope itself.
- Faraday Tomography is a computationally demanding but scientifically rich technique: The study of cosmic magnetic fields, particularly on large scales, relies heavily on Faraday Tomography. This technique, while powerful, presents significant computational hurdles that need to be overcome for SKA-scale datasets.
- Dask is a promising tool for SKA-scale data processing: The paper demonstrates that Dask can effectively parallelize complex astronomical processing tasks, offering a scalable and flexible solution for handling the massive data streams expected from the SKA.
- Prototyping with existing telescopes is essential: Using LOFAR data as a proxy for SKA data allows for the development and validation of pipelines before the SKA is fully operational, ensuring readiness for scientific discovery.
- The "cosmic web" holds secrets about magnetic fields: The research aims to probe the faint magnetic fields within the large-scale structure of the Universe, which could reveal fundamental physics and the origins of magnetism.
Key Results
The paper reports on the initial testing of a parallelized version of their SKA-like science pipeline. A significant speedup was observed: a serialised code that took 11 hours to process a single sky pointing was reduced to just 8 minutes using the parallelized Dask pipeline. Furthermore, source parameter measurements (sky coordinates and flux density) for a detected pulsar were found to be consistent with those obtained from other radio astronomy packages, validating the technical accuracy and scientific reliability of their pipeline.
Limitations and Future Work
The authors acknowledge that the current work is an ongoing development effort. They aim to have a fully working and deployable spectral imaging and Faraday Tomography pipeline by the end of 2018. Future work will involve further testing the scalability and implementation of these SKA-like science pipelines, optimizing the Faraday Tomography algorithm for SKA sensitivities, and potentially deploying these algorithms on GPUs. The ultimate goal is to enable tight constraints on the magnetised large-scale structure of the Universe and contribute to cutting-edge scientific discoveries.
Summary updated