Explainable sequence-to-sequence GRU neural network for pollution forecasting

Sara Mirzavand Borujeni, Leila Arras, Vignesh Srinivasan, Wojciech SamekScientific 2023 · DOI

TL;DR

Researchers have successfully "opened the black box" of deep learning-based air pollution forecasting by extending Layer-wise Relevance Propagation (LRP) to sequence-to-sequence GRU networks. This approach not only validates that the models are making decisions based on sound environmental physics—like temperature inversions and solar radiation cycles—but also allows for the creation of a simplified, 57% faster model architecture with negligible loss in accuracy.

What Problem Does This Paper Address?

While deep neural networks (DNNs) significantly outperform traditional regression models in forecasting air pollutants like $PM_{10}$ and $O_3$, they are notoriously opaque. In high-stakes domains like public health, this "black box" nature is a liability.

The authors highlight the risk of "Clever Hans" predictors—models that achieve high accuracy by latching onto irrelevant artifacts in the data rather than the actual underlying phenomena. To foster trust and gain scientific insights, we need to understand which input features (e.g., humidity, wind speed, or time of day) are driving specific predictions. However, explaining sequence-to-sequence (seq2seq) models with Gated Recurrent Units (GRUs) is technically challenging due to their complex gating mechanisms.

Methods

The study utilizes a sequence-to-sequence (seq2seq) architecture consisting of an encoder and a decoder, both using Gated Recurrent Units (GRUs). The model is trained to forecast four major pollutants ($PM_{10}$, $NO_2$, $NO$, $O_3$) for the next 48 hours based on the previous 7 days of history.

The core contribution is the extension of Layer-wise Relevance Propagation (LRP) to this architecture. LRP works by redistributing the prediction value from the output layer back through the network to the input features, following specific "conservation rules." To make this work for GRUs, the authors had to define how relevance flows through:

  • Linear layers: Using the LRP-epsilon rule.
  • Multiplicative gates: Using a "signal-take-all" approach where relevance is assigned to the signal rather than the gate.
  • Activations: Using identity redistribution.

They validated their LRP implementation using an arithmetic "toy task" where the ground-truth contributions were known, ensuring the heatmaps produced were mathematically sound.

Datasets

The authors used a massive dataset collected from 22 measuring stations in Germany (Stuttgart and North Rhine-Westphalia) spanning a decade (2010–2020).

  • Scale: Over 1 million data sequences.
  • Features: 22 variables including concentrations of the four pollutants, meteorological data (temp, precipitation, sunshine, humidity, wind), and temporal features (hour, day, working vs. work-free day).
  • Resolution: 1-hour intervals.

Key Insights and Ideas

  • Physics-Consistent AI: The LRP heatmaps confirmed the model learned real-world chemistry. For example, it identified high temperature and low humidity as the primary drivers for Ozone ($O_3$), which aligns with the known role of solar radiation in ozone formation.
  • Identifying Pollution Sources: At a specific station in Bielefeld, LRP identified that certain wind directions were highly relevant for high $NO$ predictions. Mapping these directions revealed they pointed directly toward a local airport and an energy production plant.
  • The "Clever Hans" Check: By visualizing the "morning peak" for $NO$ (linked to traffic) and "evening peak" for $PM_{10}$ (linked to domestic heating), the researchers confirmed the model was focusing on the correct temporal patterns rather than random noise.
  • Model Pruning via XAI: The heatmaps revealed that the historical "encoder" data (the past 7 days) was far less relevant than the immediate forecast data. This suggested the model was over-engineered for the task.

Key Results

  • Predictive Accuracy: The base model achieved a Mean Absolute Error (MAE) of 3.73 μg/m³.
  • Efficiency Gains: Based on LRP insights, the authors built a "decoder-only" model, removing the encoder and several irrelevant features (like "year" or "sunshine").
  • Performance Trade-off: The simplified model reduced training time from 17.6 hours to 7.5 hours (a ~57% speedup) while only increasing the error slightly to 4.63 μg/m³. This demonstrates that XAI can be a powerful tool for neural architecture search and optimization.

Limitations and Future Work

The authors acknowledge that while LRP identifies which features are important, it doesn't explicitly show the interaction between features (e.g., how wind and temperature work together).

Future work could involve:

  1. Global Generalization: Testing the model on datasets from different climates and urban layouts outside of Germany.
  2. Interaction Analysis: Developing methods to quantify how groups of variables interact to drive pollution events.
  3. Operational Integration: Using these explanations to help city planners identify which specific local factors (like a particular industrial site) are most responsible for air quality limit breaches.

Summary updated

Abstract

The goal of pollution forecasting models is to allow the prediction and control of the air quality. Non-linear data-driven approaches based on deep neural networks have been increasingly used in such contexts showing significant improvements w.r.t. more conventional approaches like regression models and mechanistic approaches. While such deep learning models were deemed for a long time as black boxes, recent advances in eXplainable AI (XAI) allow to look through the model's decision-making process, providing insights into decisive input features responsible for the model's prediction. One XAI technique to explain the predictions of neural networks which was proven useful in various domains is Layer-wise Relevance Propagation (LRP). In this work, we extend the LRP technique to a sequence-to-sequence neural network model with GRU layers. The explanation heatmaps provided by LRP allow us to identify important meteorological and temporal features responsible for the accumulation of four major pollutants in the air ([Formula: see text], [Formula: see text], [Formula: see text], [Formula: see text]), and our findings can be backed up with prior knowledge in environmental and pollution research. This illustrates the appropriateness of XAI for understanding pollution forecastings and opens up new avenues for controlling and mitigating the pollutants' load in the air.
BibTeX
@inproceedings{borujeni2023explainable,
  author = {Sara Mirzavand Borujeni and Leila Arras and Vignesh Srinivasan and Wojciech Samek},
  title = {{Explainable sequence-to-sequence GRU neural network for pollution forecasting}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-023-35963-2},
}
Figure