Abstract
Message importance measure (MIM) is applicable to characterize the importance of information in the scenario of big data, similar to entropy in information theory. In fact, MIM with a variable parameter can make an effect on the characterization of distribution. Furthermore, by choosing an appropriate parameter of MIM, it is possible to emphasize the message importance of a certain probability element in a distribution. Therefore, parametric MIM can play a vital role in anomaly detection of big data by focusing on probability of an anomalous event. In this paper, we propose a parameter selection method of MIM focusing on a probability element and then present its major properties. In addition, we discuss the parameter selection with prior probability, and investigate the availability in a statistical processing model of big data for anomaly detection problem.
Introduction
In the era of Big Data, identifying rare but critical events—such as fraudulent transactions or network intrusions—is akin to finding a needle in a haystack. Traditional information-theoretic measures, like Shannon entropy, are designed to characterize the overall distribution, effectively prioritizing the majority subsets. While previous iterations of the Message Importance Measure (MIM) were developed to target minority subsets, their parameter selection was often imprecise, typically focusing only on the absolute minimum probability and overlooking other significant rare anomalies. This research provides a rigorous mathematical framework for selecting MIM parameters to focus on any specific probability element of interest, enhancing its utility for surgical anomaly detection.
Methodology: Parametric MIM and Parameter Selection
The core of this framework is the Parametric MIM, defined as: $$L(\mathbf{p}, \varpi) = \log \sum_{i=1}^{n} p_i e^{\varpi(1-p_i)}$$ where $\varpi$ represents the importance coefficient. The authors propose several advancements for tuning this parameter:
- Reciprocal Parameter Selection: By setting the importance coefficient $\varpi_j = 1/p_j$, the probability element $p_j$ is established as the "Principal Component" of the measure. This ensures that the term associated with $p_j$ dominates the sum, allowing practitioners to amplify the signal of a specific rare event while suppressing background noise.
- Optimization via KKT Conditions: In cases where the exact probability of an anomaly is unknown but a range is suspected, Karush-Kuhn-Tucker (KKT) conditions are employed to find an optimal $\varpi^*$ that maximizes sensitivity.
- Statistical Convergence: Utilizing the Law of Large Numbers and the Central Limit Theorem, the study proves that the empirical MIM converges even in noisy environments, ensuring stability for real-world big data applications.
Results and Numerical Validation
The proposed method was validated using various synthetic statistical distributions, including Binomial, Poisson, Geometric, Normal, and Rayleigh distributions, with the Uniform distribution serving as a baseline.
As shown in Figure 2, the operational MIM is significantly higher for skewed distributions (like Geometric and Poisson) when focusing on rare elements compared to the uniform baseline. This demonstrates the "Focus" effect, where choosing $\varpi = 1/p_j$ allows the measure to ignore frequent events and amplify the importance of specific indices.
Figure 3 further illustrates the sensitivity of the measure to non-uniformity. Unlike Shannon entropy, which is maximized by a uniform distribution, the parametric MIM (when focused on rare events) increases dramatically for non-uniform distributions as the number of elements grows. This property makes it a superior tool for flagging unusual system states.
The research successfully proves four major properties of the parametric MIM, most notably the "Principal Component" property, which guarantees that the chosen probability element dictates the measure's behavior.
Conclusion
This paper introduces a refined method for tuning the Message Importance Measure to highlight specific anomalies in big data. By utilizing reciprocal parameter selection, MIM becomes a more surgical tool than traditional entropy. While the method currently depends on some prior knowledge of the target probability range and involves higher computational complexity due to exponential sums, it offers a robust framework for future real-time anomaly detection algorithms in high-dimensional environments.
Summary updated