Estimating Total Correlation with Mutual Information Estimators

Ke Bai, Pengyu Cheng, Weituo Hao, Ricardo Henao, Larry CarinAISTATS 2023 · OpenReview

Abstract

Total correlation (TC) is a fundamental concept in information theory that measures statistical dependency among multiple random variables. Recently, TC has shown noticeable effectiveness as a regularizer in many learning tasks, where the correlation among multiple latent embeddings requires to be jointly minimized or maximized. However, calculating precise TC values is challenging, especially when the closed-form distributions of embedding variables are unknown. In this paper, we introduce a unified framework to estimate total correlation values with sample-based mutual information (MI) estimators. More specifically, we discover a relation between TC and MI and propose two types of calculation paths (tree-like and line-like) to decompose TC into MI terms. With each MI term being bounded, the TC values can be successfully estimated. Further, we provide theoretical analyses concerning the statistical consistency of the proposed TC estimators. Experiments are presented on both synthetic and real-world scenarios, where our estimators demonstrate effectiveness in all TC estimation, minimization, and maximization tasks.

Introduction

In modern machine learning, measuring and optimizing statistical dependencies between variables is a core challenge. While Mutual Information (MI) is the standard tool for quantifying the relationship between two variables—extensively used in contrastive learning and information bottleneck frameworks—it does not naturally scale to groups of three or more variables. The multi-variable generalization of MI is known as Total Correlation (TC).

Despite its theoretical utility, TC is notoriously difficult to compute from data samples. Exact calculation typically requires knowledge of the underlying probability distributions, which are rarely available for high-dimensional neural embeddings. Previous approaches often relied on restrictive assumptions, such as Gaussianity. This work addresses these limitations by proposing a sample-based TC estimator that leverages the power of existing neural MI estimators without requiring prior knowledge of data distributions.

Methodology: TCeld Framework

The authors propose TCeld (Total Correlation Estimation via Linear Decomposition), a framework that decomposes the complex multi-variable TC into a sum of standard pairwise MI terms. This is based on a mathematical theorem showing that the TC of a set of variables can be partitioned into the TC of two disjoint subgroups plus the MI between those subgroups.

By applying this decomposition recursively, TC can be expressed entirely as a summation of MI terms. The paper explores two primary topological strategies for this decomposition:

  1. Line-like Decomposition: Variables are processed sequentially, calculating the MI between each new variable and the set of all previously processed variables.
  2. Tree-like Decomposition: The set of variables is recursively divided into two balanced halves, forming a binary tree structure.
Decomposition strategies: Tree-like (left) and Line-like (right) paths to break TC into pairwise MI terms.

Once decomposed, any off-the-shelf variational MI estimator (e.g., MINE, InfoNCE, or CLUB) can be used to estimate the individual terms. This modularity allows TCeld to inherit the statistical properties of the chosen MI estimator, such as consistency or specific bounding behaviors.

Experimental Evaluation

The framework was evaluated across synthetic benchmarks and real-world applications including disentangled representation learning and contrastive learning.

Synthetic Validation

Using 20-dimensional multivariate Gaussian data, the authors verified that TCeld accurately tracks ground-truth TC values. As shown in the results, the choice of decomposition path interacts with the underlying MI estimator. For instance, tree-like paths tend to be more stable for lower-bound estimators like InfoNCE by maintaining balanced input dimensions.

TC estimation results using various MI estimators (MINE, NWJ, InfoNCE, CLUB) comparing line-like and tree-like paths against true TC.

Disentanglement and Contrastive Learning

In practical tasks, TCeld proved highly effective:

  • TC Minimization: On the ColorMNIST dataset, minimizing TC between latent dimensions for digit, color, and style led to superior disentanglement compared to standard VAEs, achieving 96.38% digit transfer accuracy.
  • TC Maximization: The framework was applied to text contrastive learning (SimCSE/PromptBERT) to maximize agreement across four augmentations simultaneously. The resulting model, PromptRoBERTa-TC, achieved a state-of-the-art 79.58 average Spearman correlation on STS benchmarks, outperforming traditional two-augmentation baselines.

Conclusion and Future Work

TCeld provides a scalable and mathematically grounded approach to multi-variable dependency modeling. By reducing TC estimation to a series of MI estimation tasks, it unlocks the use of advanced neural estimators for complex joint optimization. While the framework inherits any biases present in the underlying MI estimators, it represents a significant step toward robust, distribution-free TC estimation. Future research will focus on the sample complexity of these decompositions and their application to ensemble learning and model distillation.

Summary updated

Abstract

Total correlation (TC) is a fundamental concept in information theory that measures statistical dependency among multiple random variables. Recently, TC has shown noticeable effectiveness as a regularizer in many learning tasks, where the correlation among multiple latent embeddings requires to be jointly minimized or maximized. However, calculating precise TC values is challenging, especially when the closed-form distributions of embedding variables are unknown. In this paper, we introduce a unified framework to estimate total correlation values with sample-based mutual information (MI) estimators. More specifically, we discover a relation between TC and MI and propose two types of calculation paths (tree-like and line-like) to decompose TC into MI terms. With each MI term being bounded, the TC values can be successfully estimated. Further, we provide theoretical analyses concerning the statistical consistency of the proposed TC estimators. Experiments are presented on both synthetic and real-world scenarios, where our estimators demonstrate effectiveness in all TC estimation, minimization, and maximization tasks.
BibTeX
@inproceedings{bai2023estimating,
  author = {Ke Bai and Pengyu Cheng and Weituo Hao and Ricardo Henao and Larry Carin},
  title = {{Estimating Total Correlation with Mutual Information Estimators}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2023},
  url = {https://openreview.net/forum?id=e2CIjUfN14},
}
Figure