The BP Dependency Function: a Generic Measure of Dependence between Random Variables

Guus Berkelmans, Joris Pries, Sandjai Bhulai, Rob van der Mei

Abstract

Measuring and quantifying dependencies between random variables (RV's) can give critical insights into a data-set. Typical questions are: 'Do underlying relationships exist?', 'Are some variables redundant?', and 'Is some target variable $Y$ highly or weakly dependent on variable $X$?' Interestingly, despite the evident need for a general-purpose measure of dependency between RV's, common practice of data analysis is that most data analysts use the Pearson correlation coefficient (PCC) to quantify dependence between RV's, while it is well-recognized that the PCC is essentially a measure for linear dependency only. Although many attempts have been made to define more generic dependency measures, there is yet no consensus on a standard, general-purpose dependency function. In fact, several ideal properties of a dependency function have been proposed, but without much argumentation. Motivated by this, in this paper we will discuss and revise the list of desired properties and propose a new dependency function that meets all these requirements. This general-purpose dependency function provides data analysts a powerful means to quantify the level of dependence between variables. To this end, we also provide Python code to determine the dependency function for use in practice.

Introduction

In modern data science, the ability to quantify the relationship between variables is fundamental. Whether performing feature selection, identifying redundancies, or exploring causal hints, analysts require a robust metric for dependency. For decades, the Pearson Correlation Coefficient (PCC) has served as the default tool. However, the PCC is fundamentally limited: it only captures linear relationships and is inherently symmetric.

Consider a simple functional relationship where $Y = X^2$. While $Y$ is perfectly dependent on $X$, the PCC may return a value near zero if the data is centered around the origin. Furthermore, existing alternatives like Mutual Information or Distance Correlation often fail to meet a consistent set of theoretical requirements. This paper introduces the Berkelmans-Pries (BP) dependency function, a metric designed to satisfy a rigorous set of eight "ideal" properties for dependency measures.

The Quest for Ideal Properties

The authors argue that for a dependency measure to be truly general-purpose, it must adhere to a strict axiomatic framework. They propose eight properties (II.1–II.8) that any such function should satisfy:

  1. Range: The measure should be scaled between 0 (representing independence) and 1 (representing full functional dependence).
  2. Asymmetry: Dependency is often directional. $Dep(Y|X)$ should not necessarily equal $Dep(X|Y)$.
  3. General Applicability: The function must handle any combination of continuous, discrete, and categorical variables.
  4. Unambiguity: Results should align with intuitive probability; for instance, if $X$ is one of $N$ independent variables picked at random, the dependency should reflect that selection probability.
  5. Invariance under Isomorphisms: Renaming categories or applying monotonic transformations should not change the dependency score.

By evaluating existing metrics against these criteria, the authors found that every major existing measure—including Spearman’s rank, Mutual Information, and Gini’s coefficient—fails at least one property. The BP function is the first to satisfy all eight.

The BP Dependency Function

The core of the BP function is the Expected Absolute Change in Distribution (UD). This approach measures how much the distribution of $Y$ changes, on average, when the value of $X$ becomes known.

Mathematical Formulation

First, the UD(X, Y) is calculated as the $L_1$ distance between the joint distribution and the product of the marginal distributions: $$UD(X, Y) = \int |P(X, Y) - P(X)P(Y)|$$

To ensure the score reaches 1 when $Y$ is a perfect function of $X$, the authors introduce a normalization step. They divide the observed change by the maximum possible change (the change in distribution when $Y$ is compared against itself): $$Dep(Y|X) = \frac{UD(X, Y)}{UD(Y, Y)}$$

This normalization creates a "relative" measure of information gain, essentially asking: "Of all the information that could possibly define $Y$, how much does $X$ provide?"

Key Insights: Asymmetry and Directionality

A defining feature of the BP function is its recognition of asymmetry. In many real-world scenarios, the flow of information is not equal in both directions. The authors use the example of DNA and eye color: while a person's eye color ($Y$) is highly dependent on their DNA ($X$), the DNA cannot be fully reconstructed from eye color alone. Thus, $Dep(Y|X)$ should be high, while $Dep(X|Y)$ should be relatively low. The BP function captures this nuance, whereas symmetric measures like PCC or Mutual Information treat both directions as identical.

Empirical Validation and Practical Estimation

While the paper is primarily theoretical, the authors validated the BP function using both synthetic mathematical edge cases and simulation studies.

  • Synthetic Tests: Using cases like $Y = X \pmod 2$, the authors proved the function correctly identifies functional dependencies and maintains asymmetry.
  • Estimation Techniques: Calculating the BP score in practice requires estimating probability distributions from finite samples. The authors compared Kernel Density Estimation (KDE) and data binning. Their simulations, involving 5,000 samples of $X \sim U(0,1)$ and $Y = X + \epsilon$, demonstrated that data binning is generally more robust and computationally efficient than KDE, particularly as dimensionality increases.

Limitations and Future Directions

Despite its theoretical strengths, the BP function faces practical challenges in estimation. The score is sensitive to "bandwidth" in KDE or the "number of bins" in data binning. Coarse binning may lose information, while overly fine binning can lead to overfitting noise. Furthermore, the empirical estimator calculated from samples tends to be slightly biased. Future research is needed to develop an asymptotically consistent, unbiased estimator and to optimize the function for very high-dimensional feature sets.

Conclusion

The Berkelmans-Pries dependency function offers a significant advancement in data analysis. By satisfying a comprehensive set of ideal properties—most notably asymmetry and general applicability—it provides a more accurate and intuitive quantification of how variables relate to one another. For the modern data analyst, it serves as a powerful alternative to the linear-only constraints of the Pearson correlation.

Summary updated

Abstract

Measuring and quantifying dependencies between random variables (RV's) can give critical insights into a data-set. Typical questions are: `Do underlying relationships exist?', `Are some variables redundant?', and `Is some target variable $Y$ highly or weakly dependent on variable $X$?' Interestingly, despite the evident need for a general-purpose measure of dependency between RV's, common practice of data analysis is that most data analysts use the Pearson correlation coefficient (PCC) to quantify dependence between RV's, while it is well-recognized that the PCC is essentially a measure for linear dependency only. Although many attempts have been made to define more generic dependency measures, there is yet no consensus on a standard, general-purpose dependency function. In fact, several ideal properties of a dependency function have been proposed, but without much argumentation. Motivated by this, in this paper we will discuss and revise the list of desired properties and propose a new dependency function that meets all these requirements. This general-purpose dependency function provides data analysts a powerful means to quantify the level of dependence between variables. To this end, we also provide Python code to determine the dependency function for use in practice.
BibTeX
@misc{berkelmans2022bp,
  author = {Guus Berkelmans and Joris Pries and Sandjai Bhulai and Rob van der Mei},
  title = {{The BP Dependency Function: a Generic Measure of Dependence between Random Variables}},
  year = {2022},
  eprint = {2203.12329},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.12329},
}
Figure