TL;DR
This paper introduces VISTA, a novel video imputation algorithm that fills in gaps in global Total Electron Content (TEC) maps, particularly over oceans, while preserving fine-scale ionospheric structures. The authors release a comprehensive TEC dataset from 2005-2020 generated using VISTA, offering a more complete and accurate view of the ionosphere for space weather research.
What Problem Does This Paper Address?
The ionosphere, a crucial layer of Earth's atmosphere, significantly impacts satellite navigation and communication systems. Total Electron Content (TEC) is a key parameter for understanding and predicting ionospheric behavior. However, existing global TEC maps suffer from large data gaps, especially over oceanic regions where ground-based receivers are scarce. Traditional methods for filling these gaps often smooth out important, smaller-scale ionospheric features, leading to an incomplete picture. This paper addresses the challenge of creating a global TEC map dataset that is both complete and retains these vital meso-scale structures.
Methods
The core of this work is the VISTA (Video Imputation with SoftImpute, Temporal smoothing and Auxiliary data) algorithm. VISTA treats TEC maps as a time series of matrices and employs a matrix factorization approach to fill missing values. It leverages observed TEC data, temporal smoothness, and auxiliary data (derived from spherical harmonics fitting) to reconstruct a complete TEC map. The algorithm is designed to preserve both large-scale patterns and localized ionospheric phenomena. The authors also detail a multi-stage data pipeline:
- Pre-processing: Outlier removal from the raw Madrigal TEC data, employing four types of outlier detection (unrealistically high values, distribution-based, region-based, and others).
- Model Fitting: Applying spherical harmonics (SH) fitting to generate auxiliary data and then using the VISTA algorithm for imputation.
- Post-processing: Smoothing the imputed TEC maps to address day-to-day boundary biases.
Datasets
The primary dataset used is the Madrigal TEC database, which provides global TEC maps with 5-minute cadence from 2005 to 2020. This database is the source of the raw, partially observed TEC data.
For validation, the authors utilize TEC measurements from JASON satellite altimeters (JASON-1, JASON-2, and JASON-3). These measurements are converted to a comparable resolution (1° latitude × 1° longitude, 5-minute cadence) to compare against the VISTA output.
The paper introduces a newly constructed complete global TEC map dataset spanning 2005-2020, generated using the VISTA algorithm on the Madrigal data. This dataset is made publicly available.
Key Insights and Ideas
- VISTA's strength in preserving meso-scale structures: Unlike traditional smoothing methods, VISTA effectively retains localized ionospheric features, which are crucial for understanding phenomena like storm-enhanced densities and polar cap patches.
- The importance of auxiliary data: Using spherical harmonics fitting to generate auxiliary data provides VISTA with a robust understanding of large-scale TEC patterns, guiding the imputation process.
- A comprehensive 16-year TEC dataset: The release of a complete TEC map dataset covering 2005-2020 is a significant contribution, providing researchers with a valuable resource for long-term ionospheric studies.
- Outlier removal is critical: The paper highlights the necessity of a rigorous outlier removal process before imputation, as even robust algorithms can be affected by erroneous data.
Key Results
- The VISTA algorithm successfully generates complete global TEC maps that align well with observed data and preserve meso-scale structures.
- The generated VISTA TEC database shows lower mean and standard deviation of bias compared to existing IGS VTEC maps when validated against JASON satellite data. For the period 2005-2007, VISTA's average bias was around -0.3 to 0.5 TECu with a standard deviation below 4 TECu, compared to IGS VTEC's average bias of 1.00 TECu and standard deviation of 4.42 TECu.
- The computational cost for a full VISTA run on a single day is estimated to be 5-20 minutes, which can be reduced to under 5 minutes by relaxing the convergence criterion slightly.
Limitations and Future Work
- Parameter tuning: While the authors provide a systematic approach to tuning VISTA parameters, they acknowledge that a more rigorous method would involve cross-validation for every single day, which is computationally intensive.
- Data availability: The quality and completeness of the VISTA dataset are inherently tied to the quality of the input Madrigal TEC data. While VISTA is robust to some outliers, extreme or systematic errors in the source data could still impact the results.
- Future work: The authors suggest that users can further screen out scattered outliers from their database if necessary, without needing to refit the VISTA model. They also point to the potential for VISTA's meso-scale preservation capability to be applied to other areas of space weather research, such as understanding plasma circulation and ionospheric-magnetospheric coupling. The release of an interactive dashboard for exploring the VISTA algorithm and data is also mentioned.
Summary updated