A Hybrid Deep Learning Model for Air Quality Prediction Based on the Time–Frequency Domain Relationship

Rui Xu, Deke Wang, Jian Li, Hang Wan, Shiming Shen, Xin GuoAtmosphere 2023 · DOI

TL;DR

This paper introduces WTformer, a novel hybrid deep learning model that combines wavelet transform with an attention-based Transformer architecture to improve air quality prediction (specifically PM2.5). By decomposing air quality data into different frequency bands, WTformer can better capture complex time-frequency relationships, leading to more accurate and interpretable predictions compared to existing models.

What Problem Does This Paper Address?

Predicting air quality is crucial for public health and environmental management. While deep learning models have shown promise, they often struggle with the inherent complexity of atmospheric data. This data is a mix of signals across various frequencies, often entangled with noise, making it difficult for standard deep learning models to extract meaningful correlations and leading to a "black box" problem where model behavior is hard to understand. This lack of interpretability hinders effective pollution control strategies. The paper aims to address these limitations by developing a model that can disentangle these frequency components and provide clearer insights into the relationships driving air quality.

Methods

The core of the proposed solution is the WTformer model, which integrates two key components:

  • Wavelet Transform (WT): This technique is used as a "frequency separator" to decompose the raw time-series data (pollutants and meteorological variables) into distinct low-frequency and high-frequency components. This process helps to disentangle the complex signals and isolate patterns related to long-term trends versus short-term fluctuations. The paper specifically uses the Stationary Wavelet Transform (SWT) with the Daubechies (db) wavelet.
  • Transformer-like Architecture (Encoder-Decoder):
    • Encoder: An improved self-attention mechanism is employed to learn correlations between different frequency bands of the input variables. This enhanced attention aims to better capture time-frequency relationships and highlight key influencing factors.
  • Decoder: This part combines Long Short-Term Memory (LSTM) for capturing temporal dependencies and another attention mechanism. The LSTM decodes the time information, and the attention network extracts relevant features from this time-decoded data to make the final PM2.5 prediction.

The authors also highlight an improved self-attention structure that addresses limitations of the original self-attention by canceling the linear layer on V and using a Tanh activation function, aiming for better interpretability and avoiding the blurring of correlations.

Datasets

The study utilizes hourly air pollutant and meteorological data collected in Guilin, China, from 2018 to 2021. This dataset includes:

  • Pollutants: PM2.5, PM10, CO, NO2, SO2, O3.
  • Meteorological Variables: Wind speed, temperature, humidity, air pressure, and rainfall.

PM2.5 is used as the target variable for prediction. The data was sourced from 10 fixed stations and 51 micro-monitoring stations within Guilin.

Key Insights and Ideas

  • Frequency Decomposition is Key: The paper strongly suggests that disentangling air quality data into different frequency bands is crucial for improving both prediction accuracy and model interpretability. Different phenomena (e.g., long-term trends vs. sudden mutations) manifest at different frequencies.
  • Time-Frequency Domain Matters: The core idea is that understanding the interplay between time and frequency is essential for accurately modeling complex environmental systems like air quality.
  • Attention for Interpretability: The improved self-attention mechanism not only helps in feature extraction but also provides a means to analyze the attention matrix, offering insights into which variables and frequency bands are most influential for PM2.5 prediction. This directly addresses the "black box" problem.
  • Hybrid Models Offer Synergy: Combining wavelet transform's signal decomposition capabilities with the powerful sequence modeling of Transformer-like architectures (specifically with LSTM in the decoder) creates a synergistic effect that outperforms individual components or simpler models.

Key Results

  • Superior Prediction Performance: The WTformer model consistently outperformed baseline models (MLP, CNN1D, GRU, Transformer, LSTM) and ablation models (SA-LA, WT-LA, LA) across various prediction time steps (1, 4, 8, 24, and 48 hours). For instance, at the 48-hour prediction step, WTformer achieved the lowest RMSE (21.379), MAE (14.943), and SMAPE (0.331).
  • Improved Sensitivity to Mutations: Ablation studies showed that removing the wavelet decomposition module (SA-LA, LA) made the model less sensitive to sudden changes or mutations in PM2.5 concentrations.
  • Reduced Lag: Models lacking the feature enhancement encoder (WT-LA, LA) exhibited a prediction lag problem, which was mitigated by the WTformer's architecture.
  • Interpretable Correlations: The attention matrix analysis revealed specific time-frequency correlations. For example, PM10, NO2, CO, and SO2 showed correlations with PM2.5 mainly in low-frequency bands, while O3 and humidity were more prominent in slower high-frequency bands. Meteorological factors like temperature and wind speed were primarily linked to PM2.5 in low-frequency bands.

Limitations and Future Work

  • Single-Site Focus: The current model is designed for a single location (Guilin). The authors acknowledge that it ignores the transmission of pollutants between stations or cities.
  • PM2.5 as a Complex Indicator: The model treats PM2.5 as a single indicator and does not fully account for its complexity or the influence of broader geographical environments.

Future research directions suggested by the authors include:

  • Multi-site Modeling: Independently modeling each station and incorporating spatial geographical factors.
  • Inter-city Pollutant Transmission: Simulating and explaining the laws of pollutant transmission between cities.
  • Reducing Prediction Error: Further refining the model to achieve even smaller prediction error ranges.

Summary updated

Abstract

Deep learning models have been widely used in time-series numerical prediction of atmospheric environmental quality. The fundamental feature of this application is to discover the correlation between influencing factors and target parameters through a deep network structure. These relationships in original data are affected by several different frequency factors. If the deep network is adopted without guidance, these correlations may be masked by entangled multifrequency data, which will cause the problem of insufficient correlation feature extraction and difficult model interpretation. Because the wavelet transform has the ability to separate these entangled multifrequency data, and these correlations can be extracted by deep learning methods, a hybrid model combining wavelet transform and transformer-like (WTformer) was designed to extract time–frequency domain features and prediction of air quality. The 2018–2021 hourly data in Guilin was used as the benchmark training dataset. Pollutants and meteorological variables in the local dataset are decomposed into five frequency bands by wavelet. The analysis of the WTformer model showed that particulate matter (PM2.5 and PM10) had an obvious correlation in the low-frequency band and a low correlation in the high-frequency band. PM2.5 and temperature had a negative correlation in the high-frequency band and an obvious positive correlation in the low-frequency band. PM2.5 and wind speed had a low correlation in the high-frequency band and an obvious negative correlation in the low-frequency band. These results showed that the laws of variables in the time–frequency domain could be found by the model, which made it possible to explain the model. The experimental results show that the prediction performance of the established model was better than that of multilayer perceptron (MLP), one-dimensional convolutional neural network (1D-CNN), gate recurrent unit (GRU), long short-term memory (LSTM) and Transformer, in all time steps (1, 4, 8, 24 and 48 h).
BibTeX
@inproceedings{xu2023hybrid,
  author = {Rui Xu and Deke Wang and Jian Li and Hang Wan and Shiming Shen and Xin Guo},
  title = {{A Hybrid Deep Learning Model for Air Quality Prediction Based on the Time–Frequency Domain Relationship}},
  booktitle = {Atmosphere},
  year = {2023},
  doi = {10.3390/atmos14020405},
}
Figure