Rainfall Forecast Model Based on the TabNet Model

Jianzhuo Yan, Tianyu Xu, Yongchuan Yu, Hongxia XuWater 2021 · DOI

TL;DR

This paper introduces a novel rainfall forecasting model by combining the TabNet deep learning architecture with advanced feature engineering and a self-supervised pre-training strategy. The approach significantly improves prediction accuracy, especially by addressing seasonal variations, and offers a robust framework for tabular data prediction tasks.

What Problem Does This Paper Address?

Accurate rainfall prediction is crucial for disaster management, flood control, and water resource planning. Traditional methods often struggle with the inherent complexity and nonlinearity of rainfall, which is influenced by numerous meteorological factors and exhibits strong seasonal patterns. While satellite observations provide valuable data, their accuracy can be affected by various environmental factors. Existing machine learning approaches have shown promise, but there's a need for models that can effectively handle tabular data, capture seasonal dynamics, and achieve higher predictive accuracy with reduced uncertainty. This paper aims to bridge this gap by developing a more robust and accurate rainfall forecasting model.

Methods

The core of this research is the application of the TabNet model, a deep learning architecture designed for tabular data that combines the interpretability of decision trees with the representation learning capabilities of deep neural networks. Key methodological contributions include:

  • TabNet Architecture: The paper leverages TabNet's unique structure, which uses an attentive transformer to perform instance-wise feature selection at each decision step, allowing the model to focus on the most relevant features for each data point.
  • Self-Supervised Pre-training: To accelerate convergence and enhance model stability, a self-supervised pre-training strategy was employed. This involved masking a significant portion of features and training the encoder-decoder architecture to predict these masked features, thereby learning robust feature representations.
  • Feature Engineering: To address the challenge of seasonal variations in rainfall, the authors implemented sophisticated feature engineering techniques. This included:
  • Feature Construction: Creating new features like Precipitable Water Vapor (PWV) and Zenith Total Delay (ZTD) by combining existing meteorological parameters and satellite-derived data. This aims to better capture atmospheric conditions relevant to rainfall.
  • Statistical Features: Generating average and standard deviation of rainfall for specific provinces, cities, and months to capture regional and temporal rainfall distributions.
  • Time-Series Feature: Calculating the average rainfall of each station over the previous 7 days to capture short-term temporal dependencies and avoid data leakage.
  • Seasonal Modeling: The data was split into "rainy season" and "non-rainy season" datasets, and separate models were trained for each to better capture the distinct characteristics of rainfall during these periods.

Datasets

The study utilized meteorological data from 26 stations located in the Beijing–Tianjin–Hebei Urban Agglomeration of China. The data spans from January 2012 to December 2016, with daily observations. The dataset includes 30 features, encompassing:

  • Geographic features: longitude, latitude, station height, city, province, station ID.
  • Time features: year, month, day.
  • Meteorological features: evaporation, surface temperature, air pressure, humidity, wind speed and direction, temperature, sunshine time, and rainfall.

The data was divided into training and testing sets, with specific splits for the rainy and non-rainy season models to ensure temporal continuity.

Key Insights and Ideas

  • TabNet's Suitability for Tabular Data: The paper strongly advocates for TabNet's effectiveness in handling tabular data, outperforming traditional deep learning architectures (like BP-NN and LSTM) and even advanced tree-based models (like LightGBM) in this domain.
  • The Power of Self-Supervised Pre-training: The self-supervised pre-training significantly improved convergence speed and model stability, demonstrating its value for tabular data tasks, not just image or text. This suggests a generalizable approach for initializing tabular models.
  • Seasonal Modeling is Crucial: Explicitly modeling rainy and non-rainy seasons separately proved vital for improving prediction accuracy, highlighting that a single model might fail to capture the distinct dynamics of these periods.
  • Feature Engineering Enhances Predictive Power: The construction of features like PWV and ZTD, along with statistical and temporal features, demonstrably improved the model's ability to capture complex rainfall patterns and extreme events.

Key Results

  • The proposed TabNet-based model (TabNet-PF) achieved superior performance compared to baseline models and traditional machine learning algorithms.
  • The MAE for the Rainy-Model (using TabNet-PF) on the test set was 0.3373, which was better than the Drier-Model (0.4825), indicating the effectiveness of seasonal modeling.
  • The TabNet with Pre-training (TabNet-P) converged faster (49 epochs vs. 58 epochs) and achieved a lower MAE (0.7403) compared to the original model.
  • In comparative experiments, the final TabNet-PF model achieved the lowest Training MAE of 0.3923 and Test MAE of 0.4099, with a high KGE of 0.88. This significantly outperformed BP-NN, LSTM, and LightGBM.

Limitations and Future Work

  • Data Imbalance: The paper notes that the "drier" season data set was more imbalanced, which could affect model performance.
  • Generalizability: While the model performed well on the Beijing–Tianjin–Hebei region, further validation on diverse geographical locations and climate zones would be beneficial.
  • Future Directions: The authors suggest exploring more data, optimizing hyperparameters further, and investigating more advanced feature engineering methods to enhance model robustness. They also propose that this approach could be a reference for similar data-mining tasks.

Summary updated

Abstract

To further reduce the error rate of rainfall prediction, we used a new machine learning model for rainfall prediction and new feature engineering methods, and combined the satellite system’s method of observing rainfall with the machine learning prediction. Based on multivariate correlations among meteorological information, this study proposes a rainfall forecast model based on the Attentive Interpretable Tabular Learning neural network (TabNet). This study used self-supervised learning to help the TabNet model speed up convergence and maintain stability. We also used feature engineering methods to alleviate the uncertainty caused by seasonal changes in rainfall forecasts. The experiment used 5 years of meteorological data from 26 stations in the Beijing–Tianjin–Hebei region of China to verify the proposed rainfall forecast model. The comparative experiment proved that our proposed method improves the performance of the model, and that the basic model used is also superior to other traditional models. This research provides a high-performance method for rainfall prediction and provides a reference for similar data-mining tasks.
BibTeX
@inproceedings{yan2021rainfall,
  author = {Jianzhuo Yan and Tianyu Xu and Yongchuan Yu and Hongxia Xu},
  title = {{Rainfall Forecast Model Based on the TabNet Model}},
  booktitle = {Water},
  year = {2021},
  doi = {10.3390/w13091272},
}
Figure