Defining the data legacy of the MAGIC telescopes: adopting a standardised data format and open-source analysis tools
Cosimo Nigro, Hyuga Abe, Shotaro Abe, J. Abhir, V. A. Acciari, I. Agudo, Tommaso Aniello, Stefano Ansoldi, L. A. Antonelli, A. Arbet Engels, Cornelia Arcaro, Manuel Artero, Katsuaki Asano, Dominik Baack, A. Babić, Andrés Baquero, U. Barres de Almeida, J. A. Barrio, Ivana Batković, Joshua Ryo Baxter, J. Becerra González, W. Bednarek, E. Bernardini, Maria-Isabel Bernardos, J. Bernete, A. Berti, Jürgen Besenrieder, C. Bigongiari, A. Biland, O. Blanch, G. Bonnoli, Ž. Bošnjak, Irene Burelli, G. Busetto, Anna Campoy-Ordaz, A. Carosi, R. Carosi, M. Carretero-Castrillo, Alberto J. Castro-Tirado, G. Ceribella, Yating Chai, A. Chilingarian, Alejo Cifuentes, S. Cikota, E. Colombo, J. L. Contreras, J. Cortina, S. Covino, Giacomo D’Amico, V. D’Elia, P. Da Vela, F. Dazzi, Alessandro De Angelis, B. De Lotto, Antonino Del Popolo, M. Delfino, J. Delgado, C. Delgado Mendez, Davide Depaoli, F. Di Pierro, L. Di Venere, D. Dominis Prester, Alice Donini, D. Dorner, M. Doro, D. Elsäesser, Gabriel Emery, Juan Escudero, Luis Fariña, Alicia Fattorini, L. Foffano, L. Font, Stefan Fröse, Satoshi Fukami, Y. Fukazawa, R. J. Garcı́a López, M. Garczarczyk, Sargis Gasparyan, M. Gaug, J. G. Giesbrecht Paiva, N. Giglietto, F. Giordano, Paweł Gliwny, N. Godinović, Roger Grau Haro, Jarred Gershon Green, D. Hadasch, A. Hahn, T. Hassan, Lea Heckmann, J. Herrera, Dario Hrupec, Moritz Hütten, Ryo Imazawa, Tomohiro Inada, Roman Iotov, Kazuma Ishio, Irene Jiménez Martínez, Jenni Jormanainen2023 · DOI
Very-high-energy (VHE, $E > 100\,{\rm GeV}$) gamma-ray telescopes have traditionally conducted science with proprietary data and software. After two decades of operations, they have accumulated thousands of hours of observations whose full scientific exploitation cannot be accomplished by the restricted group of scientists operating these instruments. The advent of a new generation of open gamma-ray observatories and, at the same time, the forthcoming end of their scientific operations, call for the realisation of public archives of their observations. With the objective to facilitate the exchange and dissemination of data from current- and next-generation gamma-ray instruments, the "Data formats for gamma-ray astronomy" (GADF) initiative was formed to provide an open and standardised format for gamma-ray astronomical data. In this contribution, we present the effort to produce data of the Major Atmospheric Gamma-ray Imaging Cherenkov (MAGIC) telescopes in this standardised format. A total of $150$ hours of observations of different sources were converted to the GADF format and then validated by analysing them with the open-source software Gammapy and comparing the results obtained against those produced with the MAGIC proprietary software, MARS. The effort to standardise and validate the Magic data in an open and shared format constitute the first fundamental milestone towards the realisation of its public data legacy.
Introduction
For the past twenty years, Very-High-Energy (VHE) gamma-ray astronomy has largely operated within "walled gardens." Instruments like the MAGIC telescopes produced data in proprietary formats that could only be analyzed using specialized, closed-source software—specifically the MARS package. This approach has historically limited the exploitation of thousands of hours of valuable observations to a small group of collaboration scientists and risks data obsolescence as MAGIC nears the end of its operational life.
As next-generation observatories like the Cherenkov Telescope Array (CTA) prepare to come online, the need for interoperable, open-source data formats has become critical. The MAGIC collaboration is addressing this by adopting the Data Formats for Gamma-ray Astronomy (GADF), a community-driven standard that ensures legacy data remains scientifically useful and compatible with modern multi-instrument analysis techniques.
Methodology
The transition from proprietary archives to a standardized public legacy involves a rigorous pipeline designed to preserve data integrity:
Data Reduction: Initial processing of low-level telescope data is handled by the legacy MARS software.
Extraction: High-level information, including event lists (reconstructed energy, direction, and arrival times) and instrument response functions (IRFs), is extracted using a dedicated MAGIC library.
Standardization: The extracted data is converted into GADF-compliant FITS files.
Validation: To ensure no information loss, the standardized data is analyzed using Gammapy (an open-source Python package) and the results are compared directly against the legacy MARS outputs.
Validation Datasets
The study validated 150 hours of MAGIC observations across three distinct scientific use cases:
Crab Nebula (50h): As the "standard candle" of gamma-ray astronomy, this bright, steady source was used to validate spectral and light curve estimations.
Mrk421 (42h): This bright but highly variable source served as a test for time-domain data handling and light curve accuracy.
M15 (57h): A dim globular cluster used to validate "upper limit" estimations, testing the software's performance in cases where signals are near the detection threshold.
Key Results and Discussion
The validation process yielded an "excellent agreement" across all test cases. Whether measuring the steady flux of the Crab Nebula, the flickering intensity of Mrk421, or the faint upper limits of M15, the results from Gammapy matched the legacy MARS results.
This success demonstrates software agnosticism: scientific results remain consistent regardless of the software used, provided the data format is standardized. This shift represents a cultural move toward "Open Science," allowing MAGIC data to be easily combined with data from other instruments, such as H.E.S.S. or the Fermi Space Telescope, for comprehensive multi-messenger studies.
Conclusion and Future Outlook
While this validation of 150 hours of data is a significant milestone, it is only the beginning. MAGIC has accumulated thousands of hours of observations over two decades. The next phase involves scaling this conversion process to the entire archive. The ultimate goal is the full public release of the MAGIC data legacy, ensuring that these two decades of observations continue to contribute to the scientific community long after the telescopes themselves cease operations.
Summary updated
Abstract
Very-high-energy (VHE, $E > 100\,{\rm GeV}$) gamma-ray telescopes have traditionally conducted science with proprietary data and software. After two decades of operations, they have accumulated thousands of hours of observations whose full scientific exploitation cannot be accomplished by the restricted group of scientists operating these instruments. The advent of a new generation of open gamma-ray observatories and, at the same time, the forthcoming end of their scientific operations, call for the realisation of public archives of their observations. With the objective to facilitate the exchange and dissemination of data from current- and next-generation gamma-ray instruments, the "Data formats for gamma-ray astronomy" (GADF) initiative was formed to provide an open and standardised format for gamma-ray astronomical data. In this contribution, we present the effort to produce data of the Major Atmospheric Gamma-ray Imaging Cherenkov (MAGIC) telescopes in this standardised format. A total of $150$ hours of observations of different sources were converted to the GADF format and then validated by analysing them with the open-source software Gammapy and comparing the results obtained against those produced with the MAGIC proprietary software, MARS. The effort to standardise and validate the Magic data in an open and shared format constitute the first fundamental milestone towards the realisation of its public data legacy.
BibTeX
@misc{nigro2023defining,
author = {Cosimo Nigro and Hyuga Abe and Shotaro Abe and J. Abhir and V. A. Acciari and I. Agudo and Tommaso Aniello and Stefano Ansoldi and L. A. Antonelli and A. Arbet Engels and Cornelia Arcaro and Manuel Artero and Katsuaki Asano and Dominik Baack and A. Babić and Andrés Baquero and U. Barres de Almeida and J. A. Barrio and Ivana Batković and Joshua Ryo Baxter and J. Becerra González and W. Bednarek and E. Bernardini and Maria-Isabel Bernardos and J. Bernete and A. Berti and Jürgen Besenrieder and C. Bigongiari and A. Biland and O. Blanch and G. Bonnoli and Ž. Bošnjak and Irene Burelli and G. Busetto and Anna Campoy-Ordaz and A. Carosi and R. Carosi and M. Carretero-Castrillo and Alberto J. Castro-Tirado and G. Ceribella and Yating Chai and A. Chilingarian and Alejo Cifuentes and S. Cikota and E. Colombo and J. L. Contreras and J. Cortina and S. Covino and Giacomo D’Amico and V. D’Elia and P. Da Vela and F. Dazzi and Alessandro De Angelis and B. De Lotto and Antonino Del Popolo and M. Delfino and J. Delgado and C. Delgado Mendez and Davide Depaoli and F. Di Pierro and L. Di Venere and D. Dominis Prester and Alice Donini and D. Dorner and M. Doro and D. Elsäesser and Gabriel Emery and Juan Escudero and Luis Fariña and Alicia Fattorini and L. Foffano and L. Font and Stefan Fröse and Satoshi Fukami and Y. Fukazawa and R. J. Garcı́a López and M. Garczarczyk and Sargis Gasparyan and M. Gaug and J. G. Giesbrecht Paiva and N. Giglietto and F. Giordano and Paweł Gliwny and N. Godinović and Roger Grau Haro and Jarred Gershon Green and Jarred Gershon Green and D. Hadasch and A. Hahn and T. Hassan and Lea Heckmann and J. Herrera and Dario Hrupec and Moritz Hütten and Ryo Imazawa and Tomohiro Inada and Roman Iotov and Kazuma Ishio and Irene Jiménez Martínez and Jenni Jormanainen},
title = {{Defining the data legacy of the MAGIC telescopes: adopting a standardised data format and open-source analysis tools}},
year = {2023},
doi = {10.22323/1.444.0827},
}