Complete Global Total Electron Content Map Dataset based on a Video Imputation Algorithm VISTA

Hu Sun, Yang Chen, Shasha Zou, Jiaen Ren, Yurui Chang, Zihan Wang, A. J. CosterScientific 2023 · DOI

TL;DR

This paper introduces VISTA, a novel video imputation algorithm that fills in gaps in global Total Electron Content (TEC) maps, particularly over oceans, while preserving fine-scale ionospheric structures. The authors release a comprehensive TEC dataset from 2005-2020 generated using VISTA, offering a more complete and accurate view of the ionosphere for space weather research.

What Problem Does This Paper Address?

The ionosphere, a crucial layer of Earth's atmosphere, significantly impacts satellite navigation and communication systems. Total Electron Content (TEC) is a key parameter for understanding and predicting ionospheric behavior. However, existing global TEC maps suffer from large data gaps, especially over oceanic regions where ground-based receivers are scarce. Traditional methods for filling these gaps often smooth out important, smaller-scale ionospheric features, leading to an incomplete picture. This paper addresses the challenge of creating a global TEC map dataset that is both complete and retains these vital meso-scale structures.

Methods

The core of this work is the VISTA (Video Imputation with SoftImpute, Temporal smoothing and Auxiliary data) algorithm. VISTA treats TEC maps as a time series of matrices and employs a matrix factorization approach to fill missing values. It leverages observed TEC data, temporal smoothness, and auxiliary data (derived from spherical harmonics fitting) to reconstruct a complete TEC map. The algorithm is designed to preserve both large-scale patterns and localized ionospheric phenomena. The authors also detail a multi-stage data pipeline:

  1. Pre-processing: Outlier removal from the raw Madrigal TEC data, employing four types of outlier detection (unrealistically high values, distribution-based, region-based, and others).
  2. Model Fitting: Applying spherical harmonics (SH) fitting to generate auxiliary data and then using the VISTA algorithm for imputation.
  3. Post-processing: Smoothing the imputed TEC maps to address day-to-day boundary biases.

Datasets

The primary dataset used is the Madrigal TEC database, which provides global TEC maps with 5-minute cadence from 2005 to 2020. This database is the source of the raw, partially observed TEC data.

For validation, the authors utilize TEC measurements from JASON satellite altimeters (JASON-1, JASON-2, and JASON-3). These measurements are converted to a comparable resolution (1° latitude × 1° longitude, 5-minute cadence) to compare against the VISTA output.

The paper introduces a newly constructed complete global TEC map dataset spanning 2005-2020, generated using the VISTA algorithm on the Madrigal data. This dataset is made publicly available.

Key Insights and Ideas

  • VISTA's strength in preserving meso-scale structures: Unlike traditional smoothing methods, VISTA effectively retains localized ionospheric features, which are crucial for understanding phenomena like storm-enhanced densities and polar cap patches.
  • The importance of auxiliary data: Using spherical harmonics fitting to generate auxiliary data provides VISTA with a robust understanding of large-scale TEC patterns, guiding the imputation process.
  • A comprehensive 16-year TEC dataset: The release of a complete TEC map dataset covering 2005-2020 is a significant contribution, providing researchers with a valuable resource for long-term ionospheric studies.
  • Outlier removal is critical: The paper highlights the necessity of a rigorous outlier removal process before imputation, as even robust algorithms can be affected by erroneous data.

Key Results

  • The VISTA algorithm successfully generates complete global TEC maps that align well with observed data and preserve meso-scale structures.
  • The generated VISTA TEC database shows lower mean and standard deviation of bias compared to existing IGS VTEC maps when validated against JASON satellite data. For the period 2005-2007, VISTA's average bias was around -0.3 to 0.5 TECu with a standard deviation below 4 TECu, compared to IGS VTEC's average bias of 1.00 TECu and standard deviation of 4.42 TECu.
  • The computational cost for a full VISTA run on a single day is estimated to be 5-20 minutes, which can be reduced to under 5 minutes by relaxing the convergence criterion slightly.

Limitations and Future Work

  • Parameter tuning: While the authors provide a systematic approach to tuning VISTA parameters, they acknowledge that a more rigorous method would involve cross-validation for every single day, which is computationally intensive.
  • Data availability: The quality and completeness of the VISTA dataset are inherently tied to the quality of the input Madrigal TEC data. While VISTA is robust to some outliers, extreme or systematic errors in the source data could still impact the results.
  • Future work: The authors suggest that users can further screen out scattered outliers from their database if necessary, without needing to refit the VISTA model. They also point to the potential for VISTA's meso-scale preservation capability to be applied to other areas of space weather research, such as understanding plasma circulation and ionospheric-magnetospheric coupling. The release of an interactive dashboard for exploring the VISTA algorithm and data is also mentioned.

Summary updated

Abstract

Ionospheric total electron content (TEC) derived from multi-frequency Global Navigation Satellite System (GNSS) signals and the relevant products have become one of the most utilized parameters in the space weather and ionospheric research community. However, there are a couple of challenges in using the global TEC map data including large data gaps over oceans and the potential of losing meso-scale ionospheric structures when applying traditional reconstruction and smoothing algorithms. In this paper, we describe and release a global TEC map database, constructed and completed based on the Madrigal TEC database with a novel video imputation algorithm called VISTA (Video Imputation with SoftImpute, Temporal smoothing and Auxiliary data). The complete TEC maps reveal important large-scale TEC structures and preserve the observed meso-scale structures. Basic ideas and the pipeline of the video imputation algorithm are introduced briefly, followed by discussions on the computational costs and fine tuning of the adopted algorithm. Discussions on potential usages of the complete TEC database are given, together with a concrete example of applying this database.
BibTeX
@inproceedings{sun2023complete,
  author = {Hu Sun and Yang Chen and Shasha Zou and Jiaen Ren and Yurui Chang and Zihan Wang and A. J. Coster},
  title = {{Complete Global Total Electron Content Map Dataset based on a Video Imputation Algorithm VISTA}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41597-023-02138-7},
}
Figure