Focusing on a Probability Element: Parameter Selection of Message Importance Measure in Big Data

Rui She, Shanyun Liu, Yunquan Dong, Pingyi Fan

Abstract

Message importance measure (MIM) is applicable to characterize the importance of information in the scenario of big data, similar to entropy in information theory. In fact, MIM with a variable parameter can make an effect on the characterization of distribution. Furthermore, by choosing an appropriate parameter of MIM, it is possible to emphasize the message importance of a certain probability element in a distribution. Therefore, parametric MIM can play a vital role in anomaly detection of big data by focusing on probability of an anomalous event. In this paper, we propose a parameter selection method of MIM focusing on a probability element and then present its major properties. In addition, we discuss the parameter selection with prior probability, and investigate the availability in a statistical processing model of big data for anomaly detection problem.

Introduction

In the era of Big Data, identifying rare but critical events—such as fraudulent transactions or network intrusions—is akin to finding a needle in a haystack. Traditional information-theoretic measures, like Shannon entropy, are designed to characterize the overall distribution, effectively prioritizing the majority subsets. While previous iterations of the Message Importance Measure (MIM) were developed to target minority subsets, their parameter selection was often imprecise, typically focusing only on the absolute minimum probability and overlooking other significant rare anomalies. This research provides a rigorous mathematical framework for selecting MIM parameters to focus on any specific probability element of interest, enhancing its utility for surgical anomaly detection.

Methodology: Parametric MIM and Parameter Selection

The core of this framework is the Parametric MIM, defined as: $$L(\mathbf{p}, \varpi) = \log \sum_{i=1}^{n} p_i e^{\varpi(1-p_i)}$$ where $\varpi$ represents the importance coefficient. The authors propose several advancements for tuning this parameter:

  1. Reciprocal Parameter Selection: By setting the importance coefficient $\varpi_j = 1/p_j$, the probability element $p_j$ is established as the "Principal Component" of the measure. This ensures that the term associated with $p_j$ dominates the sum, allowing practitioners to amplify the signal of a specific rare event while suppressing background noise.
  2. Optimization via KKT Conditions: In cases where the exact probability of an anomaly is unknown but a range is suspected, Karush-Kuhn-Tucker (KKT) conditions are employed to find an optimal $\varpi^*$ that maximizes sensitivity.
  3. Statistical Convergence: Utilizing the Law of Large Numbers and the Central Limit Theorem, the study proves that the empirical MIM converges even in noisy environments, ensuring stability for real-world big data applications.

Results and Numerical Validation

The proposed method was validated using various synthetic statistical distributions, including Binomial, Poisson, Geometric, Normal, and Rayleigh distributions, with the Uniform distribution serving as a baseline.

MIM values when the importance coefficient is tuned to different probability elements across various distributions.

As shown in Figure 2, the operational MIM is significantly higher for skewed distributions (like Geometric and Poisson) when focusing on rare elements compared to the uniform baseline. This demonstrates the "Focus" effect, where choosing $\varpi = 1/p_j$ allows the measure to ignore frequent events and amplify the importance of specific indices.

Comparison of MIM values focusing on the minimum probability element across different statistical distributions as the number of elements increases.

Figure 3 further illustrates the sensitivity of the measure to non-uniformity. Unlike Shannon entropy, which is maximized by a uniform distribution, the parametric MIM (when focused on rare events) increases dramatically for non-uniform distributions as the number of elements grows. This property makes it a superior tool for flagging unusual system states.

The research successfully proves four major properties of the parametric MIM, most notably the "Principal Component" property, which guarantees that the chosen probability element dictates the measure's behavior.

Conclusion

This paper introduces a refined method for tuning the Message Importance Measure to highlight specific anomalies in big data. By utilizing reciprocal parameter selection, MIM becomes a more surgical tool than traditional entropy. While the method currently depends on some prior knowledge of the target probability range and involves higher computational complexity due to exponential sums, it offers a robust framework for future real-time anomaly detection algorithms in high-dimensional environments.

Summary updated

Abstract

Message importance measure (MIM) is applicable to characterize the importance of information in the scenario of big data, similar to entropy in information theory. In fact, MIM with a variable parameter can make an effect on the characterization of distribution. Furthermore, by choosing an appropriate parameter of MIM, it is possible to emphasize the message importance of a certain probability element in a distribution. Therefore, parametric MIM can play a vital role in anomaly detection of big data by focusing on probability of an anomalous event. In this paper, we propose a parameter selection method of MIM focusing on a probability element and then present its major properties. In addition, we discuss the parameter selection with prior probability, and investigate the availability in a statistical processing model of big data for anomaly detection problem.
BibTeX
@inproceedings{she2017focusing,
  author = {Rui She and Shanyun Liu and Yunquan Dong and Pingyi Fan},
  title = {{Focusing on a Probability Element: Parameter Selection of Message Importance Measure in Big Data}},
  booktitle = {ICC},
  year = {2017},
  url = {https://openreview.net/forum?id=XB0H7lCZ12},
}
Figure