BERTground: A Transformer-Based Model of Background Spectra on the ISS-Based NICER Space Telescope

Anh N. Nhu, Abderahmen Zoghbi

Introduction: The Challenge of Background Noise in NICER

The Neutron star Interior Composition Explorer (NICER), located on the International Space Station (ISS), is a critical instrument for studying high-energy astrophysical phenomena like pulsars and black holes. However, NICER operates as a single-pixel detector, meaning it lacks the spatial resolution of CCD-based telescopes to distinguish between a target source and the surrounding background.

The resulting data stream is heavily contaminated by "background spectra"—noise generated by cosmic rays, solar activity, and the magnetic environment of the ISS. This noise is highly variable and can frequently obscure faint astrophysical signals. Traditional models, such as SCORPEON or 3C50, rely on rule-based systems or a limited set of variables, often failing to accurately predict high-intensity background "spikes."

The BERTground Architecture

To address these limitations, researchers developed BERTground, a deep-learning architecture that adapts Transformer-based models for tabular space-weather data. The model employs a three-stage pipeline to process 46 raw numerical variables describing the spacecraft's environment.

The BERTground pipeline, featuring Feature Group Tokenizers, a Transformer Encoder, and a Tabular ResNet.
  1. Feature Group Tokenizers: Raw inputs (e.g., ISS longitude, magnetic field strength) are grouped into five physically relevant categories, such as "Geomagnetic" and "Solar Position." These groups are processed through sigmoid-activated linear layers to create "tokens" representing physical concepts.
  2. Transformer Encoder: Four Transformer layers with eight attention heads analyze the interactions between these feature groups, capturing complex non-linear relationships.
  3. Tabular Deep Residual Network (ResNet): Instead of a standard Multi-Layer Perceptron (MLP), a ResNet with skip connections is used for the final prediction. This architecture helps retain raw feature information and ensures stable training.

The model predicts photon counts across 1,180 energy bins (0.2 to 12 keV) using a custom Mean Squared Log Error (MSLE) loss function to maintain sensitivity across all energy levels.

Data and Training

BERTground was trained on over 3 million seconds of "blank sky" observations—regions of space known to be devoid of X-ray sources. The dataset included 46 "MKF" parameters detailing the spacecraft's environment and detector settings. Data from five regions were used for training and validation, while two additional regions were reserved for testing to evaluate the model's ability to generalize to unseen parts of the sky.

Results and Performance

BERTground represents a significant advancement over existing NASA standards. While the rule-based SCORPEON model achieved an $R^2$ score of only 0.03, BERTground reached an $R^2$ of 0.99, indicating it captures nearly all variance in the background signal.

Comparison of prediction accuracy between the rule-based SCORPEON model ($R^2=0.03$) and the deep-learning BERTground model ($R^2=0.99$).

The model is particularly effective in "Extreme Background" scenarios (≥ 1000 photons/sec), where it achieved a Root Mean Squared Error (RMSE) of 25.02, compared to SCORPEON’s 36.62. This performance demonstrates that BERTground can handle the non-linear spikes that typically baffle simpler models.

Conclusion and Future Directions

By treating tabular data as a structured "language" of physical features, BERTground provides a robust solution for noise subtraction in NICER data. Future work will focus on incorporating temporal inductive bias—using LSTMs or temporal Transformers—to account for the "memory" of space weather events. Additionally, efforts are underway to integrate BERTground into HEASoft, the standard software suite for astrophysicists, making this tool available for global scientific use.

Summary updated

Abstract

The Neutron star Interior Composition Explorer (NICER) is an International Space Station (ISS)-based Space Telescope developed by NASA and devoted to the study of high-energy X-Ray sources in the universe, including but not limited to neutron stars, pulsars, and black holes in stellar systems and active galactic nuclei (AGN). One prominent problem with NICER observations is the highly variable background spectra, obscuring actual signals of astrophysical sources and negatively affecting scientific analysis of the targets. Therefore, obtaining accurate estimations of the background spectra is crucial to filter the noise and facilitate better scientific discoveries of new astronomical objects. In this paper, we propose the very first Deep Neural Network architecture to model the NICER background spectra variation using information about the spacecraft and telescope associated with each observation. In particular, we develop a BERT-based architecture with tokenizers applied to different groups of features in our tabular dataset. We also introduce an adapted Tabular Deep Residual Network architecture as the predictor following the Transformer modules in our network. We show that our model outperforms the current state-of-the-art background model developed by the NICER team in most evaluation metrics. Finally, we discuss pathways and future work for the deployment of this model on NASA’s next versions of HEASARC Software packages.
BibTeX
@inproceedings{nhu2024bertground,
  author = {Anh N. Nhu and Abderahmen Zoghbi},
  title = {{BERTground: A Transformer-Based Model of Background Spectra on the ISS-Based NICER Space Telescope}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30335},
}
Figure