Automated State Estimation for Summarizing the Dynamics of Complex Urban Systems Using Representation Learning

Maira Alvi, Tim French, Philip Keymer, Rachel Cardell-OliverAAAI 2024 · DOI

Abstract

Managing complex urban systems like Wastewater Treatment Plants (WWTPs) is a balancing act involving highly non-linear biological and chemical processes. This paper introduces UFACR (Unsupervised Feature Attention with Compact Representation), a framework that leverages unsupervised deep learning and explainable AI (XAI) to automatically identify operational states. By weighting salient sensor data through SHAP scores and compressing temporal patterns via a seq2seq encoder-decoder, the system uncovers critical biological shifts and anomalies previously obscured in high-dimensional data. Evaluation on real-world and simulated datasets shows that UFACR significantly outperforms traditional dimensionality reduction techniques, offering a robust tool for automated state estimation.

Introduction

Wastewater treatment is a critical urban service governed by complex, non-linear dynamics influenced by fluctuating influent loads and environmental conditions. While modern plants are equipped with numerous sensors, these provide only a fragmented view of the "global state"—such as whether a system is dominated by healthy algae or a problematic bacterial bloom. Traditionally, identifying these states requires labor-intensive manual analysis or complex mechanistic models. Standard machine learning clustering often struggles with the high dimensionality and temporal dependencies of sensor data. This research addresses these challenges by proposing an automated, unsupervised framework to summarize complex system dynamics into interpretable operational states.

Methodology: The UFACR Framework

The proposed UFACR framework operates in three distinct stages to transform raw sensor data into meaningful state estimations.

The UFACR framework pipeline, illustrating the flow from input time series to state estimation through feature attention and compact representation.
  1. Unsupervised Feature Attention (UFA): To overcome the "curse of dimensionality," the framework first generates "pseudo-labels" using K-means clustering. A Random Forest classifier is then trained to predict these labels. By applying SHAP (SHapley Additive exPlanations), the model identifies which sensors are most influential in defining these clusters, allowing it to weight salient features and filter out noise.
  2. Compact Representation (CR): The weighted data is processed by a seq2seq encoder-decoder model using Long Short-Term Memory (LSTM) networks. This stage learns a low-dimensional "bottleneck" representation that preserves essential temporal relationships and the order of events, which is crucial for systems with time-shifted dynamics.
  3. State Estimation: Finally, clustering is performed on this refined, compressed representation to identify distinct operational states.

Experimental Evaluation

The framework was validated using two datasets:

  • High Rate Algae Pond (HRAP): One year of real-world sensor and laboratory data from a plant in Queensland, Australia.
  • Benchmark Simulation Model 2 (BSM2): An industry-standard simulation of a full-scale plant over 609 days.

Quantitative and Qualitative Results

UFACR demonstrated substantial improvements over standard techniques such as Principal Component Analysis (PCA) and Discrete Fourier Transforms (DFT). Quantitatively, the method achieved a 39% performance gain on the HRAP dataset and a 44% gain on the BSM2 dataset in terms of Silhouette coefficients.

Comparison of clustering quality across different methods using t-SNE visualizations on HRAP and BSM2 datasets.

Qualitatively, the "blind" clusters discovered by the AI were found to map directly to physical biological realities. For example, in the HRAP data, the model successfully distinguished between "Bacterial Dominated" and "Algal Dominated" states, a distinction confirmed by domain experts. Furthermore, the model identified early warning signs for "partial nitrification" and high organic carbon loading events before they reached critical thresholds.

Discussion and Conclusion

A key strength of UFACR is its ability to preserve temporal awareness, ensuring that the sequence of events—often delayed in large-scale biological processes—is captured in the state estimation. By using XAI as a proxy task for feature selection, the model effectively bridges the gap between raw data and expert-validated insights.

While the framework shows great promise, its effectiveness is sensitive to the chosen temporal "window size," which must be tuned to the specific process. Future work will explore the transition from state summarization to proactive control, potentially using these identified states to trigger automated adjustments in plant operations. Although developed for wastewater treatment, the UFACR framework provides a generalizable approach for monitoring other complex urban infrastructures, such as power grids and traffic networks.

Summary updated

BibTeX
@inproceedings{alvi2024automated,
  author = {Maira Alvi and Tim French and Philip Keymer and Rachel Cardell-Oliver},
  title = {{Automated State Estimation for Summarizing the Dynamics of Complex Urban Systems Using Representation Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30344},
}
Figure