Square Kilometre Array Science Data Challenge 3a: foreground removal for an EoR experiment

Anna Bonaldi, Philippa Hartley, Robert D. Braun, S. J. D. Purser, Anshuman Acharya, Kyungjin Ahn, Miguel Aparicio Resco, Omkar Bait, Michele Bianco, Abhijit Chakraborty, Emma Chapman, Suman Chatterjee, K Chege, Xiao Chen, Zhong Chen, Luke Conaboy, M. Cruz, Laura Darriba, M. De Santis, Philipp Denzel, Kai Diao, Jennifer Feron, Chris Finlay, B. K. Gehlot, S. Ghosh, Sambit K. Giri, R D P Grumitt, Sungwook E. Hong, Takaaki Ito, Min Jiang, C. Jordan, Shinna Kim, Minsu Kim, J. Kim, Shreyam Parth Krishna, Akshay Kulkarni, M. López-Caniego, I. Labadie-García, H. W. Lee, David Lee, Nicole Lee, J. Line, Yu‐Rong Liu, Yi Mao, Aishrila Mazumder, Florent Mertens, S. Munshi, Ainulnabilah Nasirudin, Shaoqing Ni, Viraj Nistane, Carina Norregaard, Donald M. Null, A. R. Offringa, H. Seo, Se–Heon Oh, David Parkinson, Jonathan R. Pritchard, M. Ruiz-Granda, Valle López, Huanyuan Shan, Rohit Sharma, Cathryn M. Trott, Shintaro Yoshiura, Li Zhang, X. Zhang, Q. Zheng, Z. Zhu, Shuai Zuo, Takuya Akahori, P. Alberto, Erwan Allys, Tao An, D. Anstey, Ji‐Hye Baek, Basavraj, S. A. Brackenhoff, P. Browne, E Ceccotti, Hao Chen, Tianyue Chen, S Choudhuri, Mahbuba Choudhury, Jonathan Coles, Joseph M. Cook, David Cornu, Steven Cunnington, Sonali Das, Eloy de Lera Acedo, J.‐m. Delouis, Fēi Dèng, Junjun Ding, Khandakar Md Asif Elahi, P. Fernandez, Christian Fernandéz, A. Alcázar, V. Galluzzi, L. Q. Gao, Utpal Garain, Julián Garrido, Marie-Lou Gendron-MarsolaisMonthly 2025 · DOI

TL;DR

This paper presents the results of the SKA Science Data Challenge 3a (SDC3a), a community-wide effort to test foreground removal techniques for the Epoch of Reionization (EoR) 21cm signal using simulated SKA-Low data. The challenge revealed promising capabilities in current methods, with several teams achieving significant foreground suppression, though accurate error estimation remains a key challenge.

What Problem Does This Paper Address?

The faint 21cm signal from the Epoch of Reionization (EoR) is a crucial probe of the early Universe's structure and evolution. However, this signal is buried beneath orders of magnitude stronger foreground emissions from our own galaxy and extragalactic sources, as well as instrumental and systematic effects. Recovering the EoR signal requires sophisticated techniques to disentangle it from these contaminants. The Square Kilometre Array (SKA) will be a powerful instrument for this, but its low-frequency component (SKA-Low) presents unique challenges due to its wide field of view and sensitivity to foregrounds. This paper addresses the critical need to develop and validate methods for foreground mitigation in preparation for SKA-Low observations.

Methods

The SDC3a challenge invited teams to develop and apply their own methods for foreground removal and EoR signal recovery. The approaches varied, but common themes included:

  • Image-based methods: Using techniques like Gaussian Process Regression (GPR), Principal Component Analysis (PCA), and deep learning (e.g., U-Net, transformer-based networks) to model and subtract foregrounds from image cubes.
  • Visibility-based methods: Directly working with interferometric visibilities to model and remove foregrounds before imaging.
  • Foreground avoidance: Identifying and masking regions in parameter space (e.g., delay space) that are dominated by foregrounds.
  • Hybrid approaches: Combining multiple techniques, such as point source subtraction followed by GPR or PCA, or using deep learning for initial deconvolution before applying other methods.
  • Power spectrum estimation: Calculating the cylindrical 2D power spectrum of the recovered EoR signal, often with specific binning requirements.

Datasets

The challenge utilized a realistic, simulated SKA-Low dataset. Key features of the dataset included:

  • SKA-Low simulated data: Realistic simulations of SKA-Low observations between 106 MHz and 196 MHz.
  • Foreground contamination: Included extragalactic and Galactic emission, instrumental effects, and systematic errors.
  • EoR signal: A simulated EoR signal embedded within the foregrounds.
  • Test dataset: A separate dataset provided for teams to validate their pipelines, including a known EoR model.
  • "True" EoR signal: The ground truth EoR signal was provided for scoring and analysis.

Key Insights and Ideas

  • Diverse approaches, converging on success: While teams employed a wide range of techniques, many achieved significant foreground suppression, indicating the field is progressing well.
  • Deep learning shows promise: Several teams leveraged deep learning architectures (U-Nets, transformers) for tasks like deconvolution and foreground modeling, showing promising results.
  • GPR and PCA remain workhorses: Established methods like Gaussian Process Regression and Principal Component Analysis were widely used and effective, often in combination with other techniques.
  • Foreground avoidance vs. removal trade-offs: Teams explored both strategies, with removal methods generally showing higher scores, but avoidance can be less prone to introducing biases.
  • Error estimation is a major hurdle: Accurately quantifying the uncertainty in the recovered EoR signal proved to be a significant challenge for most teams, with true errors often being underestimated.
  • Reproducibility is key: The challenge highlighted the importance of well-documented and shareable code, with reproducibility badges awarded to teams demonstrating best practices.

Key Results

  • Foreground suppression: Most teams successfully reduced foreground contamination by at least one order of magnitude.
  • EoR signal recovery: Seven teams achieved a median error in the EoR power spectrum recovery below the true signal.
  • Smallest residual: The smallest residual overall was $4.2_{-4.2}^{+20} \times 10^{-4}\, \rm {K}^2h^{-3}$cMpc$^{3}$.
  • Error estimation accuracy: The most accurate error bars accounted for approximately $60 \pm 20\%$ of the true errors.
  • Top performers: The DOTSS-21cm team (with Advanced ML-GPR) and HIMALAYA team achieved the best overall scores.

Limitations and Future Work

  • Systematic errors: The paper acknowledges that the simulation, while realistic, may not capture all real-world systematic errors (e.g., polarization leakage).
  • Error estimation: The significant challenge in accurately estimating confidence levels is a major area for future work. Teams need to develop more robust methods for propagating systematic uncertainties.
  • Pipeline development: Many teams are still refining their pipelines, and future work will focus on improving specific steps like point source subtraction, diffuse foreground modeling, and incorporating instrumental effects more accurately.
  • Comparison of pipelines: The challenge demonstrated the value of comparing independent pipelines to identify residual biases and improve methods. This collaborative approach is seen as crucial for future progress.

Summary updated

Abstract

ABSTRACT We present and analyse the results of the Science Data Challenge 3a (SDC3a, https://sdc3.skao.int/challenges/foregrounds), an epoch of reionization (EoR) foreground-removal exercise organized by the Square Kilometre Array Observatory (SKAO) on SKA simulated data. The challenge ran for 8 months, from 2023 March to October. Participants were provided with realistic simulations of SKA-Low data between 106 and 196 MHz, including foreground contamination from extragalactic and Galactic emission, instrumental, and systematic effects. They were asked to deliver cylindrical power spectra of the EoR signal, cleaned from all corruptions, and the corresponding confidence levels. Here, we describe the approaches taken by the 17 teams that completed the challenge, and we assess their performance using different metrics. The challenge results provide a positive outlook on the capabilities of current foreground-mitigation approaches to recover the faint EoR signal from SKA-Low observations. The median error committed in the EoR power spectrum recovery is below the true signal for seven teams, although in some cases, there are some significant outliers. The smallest residual overall is $4.2_{-4.2}^{+20} \times 10^{-4}\, \rm {K}^2h^{-3}$cMpc$^{3}$ across all considered scales and frequencies. The estimation of confidence levels provided by the teams is overall less accurate, with the true error being typically underestimated, sometimes very significantly. The most accurate error bars account for $60 \pm 20$ per cent of the true errors committed. The challenge results provide a means for all teams to understand and improve their performance. This challenge indicates that the comparison between independent pipelines could be a powerful tool to assess residual biases and improve error estimation.
BibTeX
@inproceedings{bonaldi2025square,
  author = {Anna Bonaldi and Philippa Hartley and Robert D. Braun and S. J. D. Purser and Anshuman Acharya and Kyungjin Ahn and Miguel Aparicio Resco and Omkar Bait and Michele Bianco and Abhijit Chakraborty and Emma Chapman and Suman Chatterjee and K Chege and Xiao Chen and Zhong Chen and Luke Conaboy and M. Cruz and Laura Darriba and M. De Santis and Philipp Denzel and Kai Diao and Jennifer Feron and Chris Finlay and B. K. Gehlot and S. Ghosh and Sambit K. Giri and R D P Grumitt and Sungwook E. Hong and Takaaki Ito and Min Jiang and C. Jordan and Shinna Kim and Minsu Kim and J. Kim and Shreyam Parth Krishna and Akshay Kulkarni and M. López-Caniego and I. Labadie-García and H. W. Lee and David Lee and Nicole Lee and J. Line and Yu‐Rong Liu and Yi Mao and Aishrila Mazumder and Florent Mertens and S. Munshi and Ainulnabilah Nasirudin and Shaoqing Ni and Viraj Nistane and Carina Norregaard and Donald M. Null and A. R. Offringa and H. Seo and Se–Heon Oh and David Parkinson and Jonathan R. Pritchard and M. Ruiz-Granda and Valle López and Huanyuan Shan and Rohit Sharma and Cathryn M. Trott and Shintaro Yoshiura and Li Zhang and X. Zhang and Q. Zheng and Z. Zhu and Shuai Zuo and Takuya Akahori and P. Alberto and Erwan Allys and Tao An and D. Anstey and Ji‐Hye Baek and Basavraj and S. A. Brackenhoff and P. Browne and E Ceccotti and Hao Chen and Tianyue Chen and S Choudhuri and Mahbuba Choudhury and Jonathan Coles and Joseph M. Cook and David Cornu and Steven Cunnington and Sonali Das and Eloy de Lera Acedo and J.‐M. Delouis and Fēi Dèng and Junjun Ding and Khandakar Md Asif Elahi and P. Fernandez and Christian Fernandéz and A. Alcázar and V. Galluzzi and L. Q. Gao and Utpal Garain and Julián Garrido and Marie-Lou Gendron-Marsolais},
  title = {{Square Kilometre Array Science Data Challenge 3a: foreground removal for an EoR experiment}},
  booktitle = {Monthly},
  year = {2025},
  doi = {10.1093/mnras/staf1466},
}
Figure