TL;DR
This paper introduces the Wavelet Transform-based Multilayer Perceptron (WTMP), a novel deep learning model for forecasting air pollutant concentrations. WTMP effectively captures complex temporal patterns by combining wavelet decomposition with an enhanced Multilayer Perceptron, achieving superior accuracy and significantly faster inference times compared to state-of-the-art Transformer-based models.
What Problem Does This Paper Address?
Accurately forecasting air pollutant concentrations is crucial for environmental protection and public health. However, the dynamic nature of atmospheric pollutants, their complex interdependencies, and the influence of various covariates make this a challenging task. Traditional time-series models often struggle with non-stationarity and nonlinear relationships, while advanced deep learning models like Transformers, despite their power, can be computationally expensive and may overlook crucial temporal ordering. This paper aims to bridge this gap by developing a more efficient and accurate model for long-term air pollutant forecasting.
Methods
The core of the WTMP model lies in its hybrid architecture:
- Wavelet Decomposition (MODWT): The paper utilizes the Maximal Overlap Discrete Wavelet Transform (MODWT) with the Haar wavelet basis. This technique decomposes the pollutant time series into multiple frequency components (detail coefficients) and a smooth trend component (scaling coefficient). This decomposition helps to extract non-stationarity and nonlinear dependencies that might be missed by standard time-series models.
- Enhanced Multilayer Perceptron (MLP): The decomposed components are then fed into an improved MLP. This MLP architecture incorporates residual blocks, which are known to improve training efficiency and prevent vanishing gradients in deep networks.
- Covariate Integration: Static covariates, such as time and date information, are also incorporated. These are first dimensionality-reduced using a residual block and then concatenated with the wavelet-decomposed components before being processed by further residual blocks.
- Global Residual Connection: A final global residual connection is added, similar to the DLinear model, to map the original data and add it to the predicted output, theoretically ensuring performance no worse than DLinear.
The key innovation is the combination of wavelet decomposition for feature extraction and an efficient MLP for prediction, avoiding the complex self-attention mechanisms of Transformers.
Datasets
The model was trained and validated using atmospheric pollutant data collected from an environmental monitoring station in the Economic Development Zone of Ruian City, Zhejiang Province, China. The dataset includes various atmospheric pollutants (nitrogen oxides, sulfur oxides, carbon oxides, ozone, PM10, PM2.5) and meteorological indicators (wind speed, wind direction, temperature, relative humidity). The data spans from January 1, 2021, to June 30, 2022, with hourly sampling, totaling 13,095 data points. Data imputation was performed using the Generative Adversarial Imputation Network (GAIN) to handle missing values.
Key Insights and Ideas
- Wavelets for Non-stationarity: The paper highlights the effectiveness of wavelet decomposition in capturing the non-stationary and nonlinear characteristics inherent in air pollutant time series, which are crucial for accurate forecasting.
- MLP's Efficiency: By leveraging an enhanced MLP architecture with residual blocks, the WTMP model achieves high accuracy without the computational overhead of complex self-attention mechanisms found in Transformer models. This makes it significantly faster for both training and inference.
- Simplicity Can Be Powerful: The success of WTMP, building on the insights of DLinear, suggests that for certain time-series forecasting tasks, simpler architectures combined with effective feature engineering (like wavelet decomposition) can outperform more complex models.
- Covariates as "Highways": The strategic inclusion of covariates, especially future prediction sequence information, acts as a "highway" to prevent the loss of crucial information during the prediction process.
Key Results
- Superior Accuracy: WTMP consistently outperformed Transformer-based models (Transformer, Informer, Pyraformer, Autoformer) and LSTM across various forecast horizons. For example, in the 720/168 setting (720 hours look-back, 168 hours forecast), WTMP achieved an MSE of 0.526, significantly lower than other models.
- Significant Speedup: WTMP was reported to be more than five times faster in both inference and training compared to the best Transformer-based model.
- Reduced Error: The model demonstrated a Mean Squared Error reduction of over 15% compared to existing neural network benchmarks.
- Ablation Study Findings: The ablation study confirmed the importance of wavelet decomposition, with models lacking this feature (Model A) performing the worst. Different wavelet bases (db4, coif4, fk4) showed varying impacts, with coif4 rivaling Haar in some instances.
Limitations and Future Work
- Generalization to Other Regions: The authors acknowledge that the current study is based on data from a single location. Future work will involve testing the WTMP model on pollutant data from other regions to verify its generalization ability.
- Spatiotemporal Modeling: The paper suggests developing an atmospheric pollutant concentration model that incorporates spatiotemporal data from multiple monitoring stations, which could capture more complex spatial dependencies.
- Model Limitations: While WTMP shows strong performance, the paper implicitly suggests that for extremely complex, highly non-linear relationships or when dealing with entirely novel pollutants, further advancements might be needed. The paper also notes that while DLinear is simple and effective, it struggles with non-linear dependencies, a limitation WTMP aims to address.
Summary updated