Abstract
Uncertain data streams have been widely generated in many Web applications. The uncertainty in data streams makes anomaly detection from sensor data streams far more challenging. In this paper, we present a novel framework that supports anomaly detection in uncertain data streams. The proposed framework adopts an efficient uncertainty pre-processing procedure to identify and eliminate uncertainties in data streams. Based on the corrected data streams, we develop effective period pattern recognition and feature extraction techniques to improve the computational efficiency. We use classification methods for anomaly detection in the corrected data stream. We also empirically show that the proposed approach shows a high accuracy of anomaly detection on a number of real datasets.
Introduction
In the domains of IoT and medical monitoring, data streams are frequently characterized by uncertainty and pseudoperiodicity. Uncertainty stems from sensor malfunctions, battery depletion, or signal drops, leading to missing or erroneous data. Pseudoperiodicity refers to signals, such as electrocardiograms (ECG), that repeat cyclically but with inherent variations in duration and amplitude. The primary challenge lies in distinguishing between transient data noise and genuine anomalies, such as medical emergencies. This paper introduces a robust four-stage pipeline designed to clean uncertain data through geometric correction and intelligently segment it into comparable periods for accurate classification.
The Proposed Framework
The framework, illustrated below, integrates uncertainty processing with advanced segmentation to enhance detection performance.
- Uncertainty Identification and Correction (UICC): Instead of processing every raw data point, the system identifies "key points" (inflexions). If a key point is missing due to sensor error, the system utilizes the geometric angles between surrounding points to mathematically reconstruct the missing peak or valley.
- Time Series Compression (TSCC): To optimize computational resources, the Douglas-Peucker algorithm simplifies the raw stream into essential segments while preserving the critical morphological features of the signal.
- Period Segmentation and Summarization (PSSC): This stage represents each turning point as a Feature Vector (FV) capturing amplitude and temporal changes relative to neighbors. It employs k-means++ clustering and Silhouette values to automatically identify stable markers for segmenting the stream into individual cycles.
- Classification and Anomaly Detection (CADC): Each segmented cycle is summarized into a set of statistical features (e.g., min/max/mean amplitude, duration) and processed by classifiers like Random Forest to label the cycle as "Normal" or "Abnormal."
Experimental Evaluation
The framework was rigorously tested using eight real-world ECG datasets from the PhysioNet repository, including the MIT-BIH Arrhythmia Database (mitdb) and the Supraventricular Arrhythmia Database (svdb). These benchmarks contain various labeled anomalies, such as premature ventricular contractions (PVCs), which are critical for evaluating detection sensitivity.
Key Results
- High Accuracy and Sensitivity: Across major datasets, the framework consistently achieved classification accuracy and sensitivity scores above 90%, with some datasets reaching near 100% sensitivity.
- Robust Segmentation: The proposed Feature Vector (FV) method significantly outperformed traditional peak-based segmentation, providing higher precision and fewer false positives in noisy environments.
- Resilience to Data Loss: The UICC component proved highly stable; even with significant iterative data deletion, the system's ability to identify correct breakpoints remained nearly perfect.
Conclusion and Future Work
This research demonstrates that focusing on geometric turning points and contextual feature vectors allows for resilient anomaly detection in noisy, repeating data streams. While the Silhouette-based evaluation adds some computational overhead, it ensures high segmentation quality. Future research will aim to detect global anomalies across multiple cycles and optimize the framework for real-time deployment on low-power edge devices.
Summary updated