Abstract
In large-scale sensor networks, data redundancy often masks the underlying logical structures of the system. This paper presents an unsupervised algorithm designed to automatically discover salient correlations between sensors without prior knowledge of network topology. By employing hash-table-based data compression and an event-driven pruning strategy, the proposed method efficiently identifies significant relationships, such as the predictive power of temperature over humidity, enabling potential energy savings and reduced sensor redundancy.
Introduction
Modern industrial and residential environments increasingly rely on massive, unstructured sensor networks. However, these networks frequently produce redundant data, where multiple sensors capture overlapping information. Identifying these "salient correlations" is critical for optimizing network performance and reducing energy consumption. Traditional data mining approaches, such as the Apriori algorithm, often fail in this context due to the high computational cost of evaluating all possible sensor combinations and the inherent sparsity of event-driven data.
Methodology
The proposed approach shifts the focus from complete pattern mining to the discovery of valuable, actionable correlations. The methodology is built upon three core pillars:
Hash-Table Compression
To handle the volume of data, the algorithm utilizes a hash-table-based compression technique. Instead of processing raw time-series data, it stores unique sets of concurrent events as keys, with their frequencies as values. In empirical tests, this method reduced over 80,000 records into fewer than 40 unique patterns, significantly lowering memory overhead.
Relationship Classification
Sensor interactions are categorized into three distinct types to better understand their logical dependencies:
- Full-Contained: Event A is a subset of Event B; A always occurs when B occurs.
- Part-containing: Events A and B share a common subset of triggers but also occur independently.
- Disjoint: Events that exhibit no statistical dependency.
Pruning and Optimization
To ensure linear scalability, a mathematical pruning rule is applied. If the correlation between two small sets of events is below a certain threshold, the algorithm automatically bypasses all larger supersets containing those events. This drastically reduces the search space and allows the system to scale with the number of sensors.
Experimental Evaluation
The algorithm was validated using two primary datasets:
- A real-world deployment of over 200 sensors (acoustic, light, motion, temperature, $CO_2$, and humidity) comprising 83,979 records.
- The 2007 Intel Lab Data benchmark to verify known physical correlations, such as the relationship between temperature and battery voltage.
Results and Discussion
The hash-table approach demonstrated remarkable efficiency, representing two months of multi-sensor data with just 39 patterns. A key discovery was the strong correlation between temperature ($t$) and humidity ($h$), suggesting that humidity sensors could be omitted in favor of predictive modeling based on temperature data.
As shown in the distribution of correlation strengths, the algorithm effectively separates significant patterns from noise. The pruning effect becomes more pronounced as the confidence threshold increases, leading to significant speedups in processing time.
While the algorithm is highly time-efficient, it remains sensitive to the definition of "events" (e.g., specific percentage changes in readings). Future work will focus on addressing memory constraints in extremely high-dimensional, non-sparse datasets and exploring dynamic correlations that evolve over time.
Conclusion
This research provides a scalable, unsupervised framework for discovering hidden dependencies in sensor networks. By prioritizing "valuable" patterns over exhaustive mining, the method offers a practical solution for engineers looking to streamline large-scale deployments and reduce operational costs.
Summary updated