TL;DR
This paper introduces a novel method for identifying periods of inflow and infiltration (I/I) in sewer systems by leveraging time-series anomaly detection on in-sewer data alone. This approach bypasses the need for external data like rainfall, offering improved accuracy and generalizability for I/I diagnosis.
What Problem Does This Paper Address?
Urban sewer systems are critical for public health and environmental protection. However, they are susceptible to inflow and infiltration (I/I), where external water (from rain, snowmelt, or groundwater) enters the system. This I/I can lead to significant problems like flooding, water quality degradation, and increased strain on wastewater treatment plants. Accurately identifying when I/I occurs is a crucial first step for managing these issues, but current methods often rely on external data (like rainfall) and require manual adjustments, leading to inaccuracies and limited applicability across different conditions. This paper aims to solve this by developing a robust, data-driven method that can reliably distinguish between periods with and without I/I using only data collected directly from the sewer network.
Methods
The core of the proposed method is an iterative anomaly detection approach. It begins by training a time-series reconstruction model (specifically, the Prophet model) to learn the "normal" behavior of sewer data (flow, temperature, conductivity) during dry weather conditions, when I/I is absent. This model captures trends, periodic variations (daily, seasonal), and exceptional events.
The key innovation lies in how it identifies I/I periods:
- BWF Reconstruction: A model is trained to represent the Base Wastewater Flow (BWF) – the expected flow without I/I.
- Anomaly Detection: The model then identifies deviations from this reconstructed BWF as anomalies. These anomalies are flagged as potential I/I events. The method uses the k-sigma principle, where data points falling outside a certain range (mean ± k * standard deviation) of the residual term are considered anomalous.
- Iterative Refinement: To improve accuracy, the process is iterative. Identified anomalies (potential I/I periods) are removed, and the BWF reconstruction model is retrained. This loop continues until the model's residuals meet normality tests, the maximum residual is below a threshold, or the correlation between successive reconstructions stabilizes.
- Isolated Point Removal: A final step smooths out short, isolated misclassifications by using a sliding window and majority voting.
This approach is novel because it relies solely on in-sewer measurements, eliminating the need for external data and manual parameter tuning.
Datasets
The paper utilizes two types of datasets:
- Simulated Data: Generated from a detailed sewer network model of a coastal Australian city. This data was used to test the algorithm's performance under three distinct scenarios: dry weather-dominated, 50/50 wet/dry weather, and wet weather-dominated. This allows for precise ground-truth comparisons.
- Real-life Data: Hourly flow data collected from the inlet of a wastewater treatment plant in Australia over a five-month period (September 2022 to February 2023). Rainfall data from the Bureau of Meteorology (at one-minute intervals) was collected for comparative analysis but not used by the algorithm itself.
Key Insights and Ideas
- Data-Driven Independence: The most significant insight is the successful demonstration that I/I periods can be reliably identified using only in-sewer data. This liberates the diagnostic process from the complexities and limitations of relying on external factors like rainfall, which have inherent delays and variable impacts.
- Anomaly Detection as a Unifier: The paper shows that treating I/I as an "anomaly" in time-series data provides a unified framework that can capture I/I from various sources (rainfall, snowmelt, seawater intrusion) without needing separate detection logic for each.
- Iterative Refinement is Key: The iterative nature of the BWF reconstruction and anomaly detection is crucial. It allows the model to progressively "learn" the true dry-weather patterns by shedding the influence of I/I events during retraining, leading to more accurate baseline estimations.
- Flow Data Dominance: While temperature and conductivity can be used, flow data appears to be the most direct and effective indicator for I/I detection due to its immediate and pronounced response to external water ingress.
Key Results
- High Accuracy: The proposed "Auto" method achieved accuracy rates exceeding 90% across various scenarios, significantly outperforming existing manual methods (Manual 1, 2, and 3) which often fell below 80%.
- Improved Precision and Recall: The Auto method demonstrated superior precision and recall, meaning it was better at correctly identifying I/I periods and minimizing false positives and negatives. This was particularly evident in the 50/50 wet/dry scenario.
- Reduced Detection Errors: The Auto method significantly reduced errors in identifying the start and end times of I/I events, with errors as low as 0-5 minutes compared to several hours for other methods when analyzing a specific rainfall-induced I/I event.
- Correlation with Rainfall: For real-world data, the identified wet periods and total I/I amounts showed strong correlations (R² values of 0.56-0.66) with total rainfall amount and duration, providing independent validation of the algorithm's accuracy, even though rainfall data was not used by the algorithm.
- Computational Efficiency: Despite the iterative nature, the analysis of five months of hourly data on a standard computational setup was completed in just 2 minutes, indicating manageable computational load.
Limitations and Future Work
- Computational Load: While manageable, the iterative reconstruction process incurs a higher computational load compared to simpler methods. The authors suggest optimization through parallel computing or simplifying iterative steps for very large datasets.
- Data Availability: The authors state they do not have permission to share the data used in the study.
- Future Work: The paper suggests that combining multiple in-sewer variables (like flow, temperature, and conductivity) using an anomaly score (AS) could further improve performance by leveraging the advantages of different indicators. Further optimization of the iterative process and exploration of parallel computing are also implied.
Summary updated