Abstract
In many applications (e.g., anomaly detection and security systems) of smart cities, rare events dominate the importance of the total information of big data collected by Internet of Things (IoTs). That is, it is pretty crucial to explore the valuable information associated with the rare events involved in minority subsets of the voluminous amounts of data. To do so, how to effectively measure the information with importance of the small probability events from the perspective of information theory is a fundamental question. This paper first makes a survey of some theories and models with respect to importance measures and investigates the relationship between subjective or semantic importance and rare events in big data. Moreover, some applications for message processing and data analysis are discussed in the viewpoint of information measures. In addition, based on rare events detection, some open challenges related to information measures, such as smart cities, autonomous driving, and anomaly detection in IoTs, are introduced which can be considered as future research directions.
Introduction: The Challenge of Rare Events in Big Data
In the era of the Internet of Things (IoT) and Big Data, we are increasingly overwhelmed by information. However, for high-stakes applications such as autonomous driving, structural health monitoring, and cyber-security, the vast majority of collected data is "normal" and relatively uninteresting. The real value lies in rare events—the "needles in the haystack"—such as a sudden pedestrian in the road, a fraudulent transaction, or a sensor indicating a bridge failure.
Traditional information theory, pioneered by Claude Shannon, focuses on "uncertainty" or entropy. While rare events have high self-information in Shannon's sense, they are often "averaged out" when looking at the entropy of a whole system. This research addresses the need for a mathematical framework that treats importance as a primary metric, allowing for the design of systems that specifically prioritize the detection, compression, and transmission of these critical outliers.
Quantifying Importance: Beyond Shannon Entropy
To move beyond logarithmic measures of uncertainty, this paper surveys theories that quantify "subjective" or "semantic" importance through exponential amplification. Key frameworks include:
- Message Importance Measure (MIM): Unlike Shannon entropy, MIM uses an exponential operator ($e^{1-p}$). This causes the importance value to skyrocket as the probability ($p$) of an event approaches zero, effectively amplifying rare events.
- Non-parametric MIM (NMIM): A more robust iteration that eliminates the need for manual parameter tuning, making it more applicable to diverse, high-velocity big data streams.
- Message Identification (MI) Divergence: A sensitivity-enhanced metric for measuring the "distance" between probability distributions. MI divergence is significantly more effective than standard Kullback-Leibler (KL) divergence when trying to distinguish a normal state from one containing a rare anomaly.
Key Insights and Applications
The "Atypical" Set
Traditional communication focuses on the "typical set" (the most likely outcomes) to maximize efficiency. This research argues that for big data analytics, the focus must shift to the atypical set—the outliers that carry the most semantic weight. By switching from logarithmic to exponential functions, the mathematical weight of a rare event can be scaled to match its real-world impact.
Application Scenarios
- Urban Monitoring: Detecting "Urban Black Holes" where traffic inflows significantly exceed outflows, indicating potential congestion or safety issues.
- Autonomous Driving: Prioritizing obstacle detection and lane-departure data over routine environmental telemetry.
- IoT Security: Using MI Divergence to identify cyber-attacks in network logs where the signature of the attack is nearly identical to normal traffic but differs in the statistical "tails."
Future Challenges and Directions
Despite the theoretical advancements in MIM, several hurdles remain for real-world implementation:
- Latency vs. Accuracy: Rare event detection must be near-instantaneous (e.g., for autonomous braking), yet complex information measures can be computationally expensive.
- Noise vs. Rarity: Distinguishing between a critical rare event and random sensor noise remains a major challenge in messy IoT environments.
- Importance-Aware Data Cleaning: There is a need for data cleaning techniques that do not inadvertently scrub away the rare events the system is designed to find.
- Causality: Future research must better map how a rare event in one part of a smart city (e.g., a power surge) triggers rare events in another (e.g., a traffic light failure).
Conclusion
Measuring the importance of small probability events is fundamental to the next generation of smart city and IoT applications. By transitioning from uncertainty-based entropy to importance-based measures like MIM and NMIM, engineers can build more resilient and sensitive systems capable of identifying critical anomalies within the vast sea of big data.
Summary updated