Abstract
Mutual Information (MI) is a fundamental measure of statistical dependency but is notoriously difficult to estimate in high-dimensional spaces due to the high variance of traditional variational bounds. This paper introduces neural methods for estimating Point-wise Dependency (PD), focusing on instance-level relationships rather than aggregate averages. We propose two novel estimators—the Probabilistic Classifier and Density-Ratio Fitting—which leverage the stability of classification and least-squares objectives. Our results show that these methods provide more reliable MI estimates and improve performance in downstream tasks such as self-supervised representation learning and cross-modal retrieval.
Introduction
Mutual Information (MI) has long served as the primary metric for quantifying the relationship between random variables. However, as an aggregate statistic, MI only describes the average dependency across distributions. It fails to capture the strength of association between specific pairs of events, such as a particular image and its corresponding caption. Furthermore, standard neural MI estimators, such as MINE or NWJ, often suffer from significant training instability and high variance because their objective functions involve logarithms and exponentials of neural network outputs.
In "Neural Methods for Point-wise Dependency Estimation," researchers propose a shift in focus toward Point-wise Dependency (PD). Defined by the density ratio $r(x, y) = p(x, y) / (p(x)p(y))$, PD provides a granular, instance-level understanding of data relationships. By bringing the flexibility of deep learning to PD estimation, the authors develop methods that are more stable, scalable, and informative than traditional aggregate estimators.
Proposed Neural Estimators
The authors introduce two primary data-driven approaches for PD estimation that leverage the representational power of neural networks:
- Probabilistic Classifier (PC): This method reformulates dependency estimation as a binary classification task. A neural network is trained to distinguish between "real" pairs (sampled from the joint distribution $p(x, y)$) and "fake" pairs (sampled from the product of marginals $p(x)p(y)$). Using a standard cross-entropy loss, this approach benefits from the inherent stability of classification objectives.
- Density-Ratio Fitting (DRF): This approach directly minimizes the least-square difference between the true and estimated PD. Critically, the objective function is "log-free," avoiding the numerical instabilities and gradient explosions associated with the logarithms and exponentiations found in variational MI bounds.
Benchmarking and MI Estimation
While the primary goal is PD estimation, these ratios can be averaged to compute aggregate MI. In benchmarking experiments using correlated Gaussians (both linear and non-linear "Cubic" tasks), the proposed methods demonstrated significantly lower variance and bias compared to established baselines. The stability of Density-Ratio Fitting is particularly evident in high-dependency regimes where traditional bounds often fail to converge.
Applications in Representation Learning
The utility of PD estimation extends directly to self-supervised learning (SSL). The authors developed Density-Ratio Fitting Coding (D-RFC), a training objective based on their DRF estimator. When evaluated on MNIST and CIFAR-10 datasets, D-RFC consistently outperformed the popular Contrastive Predictive Coding (CPC) baseline. This performance advantage held true across both shallow and deep neural architectures, suggesting that more stable dependency estimation leads to higher-quality learned representations.
Cross-Modal Retrieval and Dataset Debugging
Beyond SSL, the Probabilistic Classifier was applied to an audio-text corpus for cross-modal retrieval. The model achieved a 96.24% top-1 accuracy in matching unsupervised word features from Word2Vec (text) and Speech2Vec (audio).
A unique advantage of PD estimation is its ability to perform "dataset debugging." Because PD operates at the instance level, it can identify adversarial or noisy samples—pairs that should be related but exhibit low point-wise dependency. This allows researchers to pinpoint mislabeled data or low-quality samples that aggregate statistics would otherwise obscure.
Conclusion
By shifting the focus from aggregate Mutual Information to instance-level Point-wise Dependency, this work provides a more stable and informative framework for understanding high-dimensional data relationships. While the performance remains dependent on neural architecture choices, the proposed methods offer a significant step forward in making dependency estimation a robust tool for deep learning. Future research may explore privacy-aware versions of these estimators to mitigate the risks of detecting sensitive attributes in anonymized datasets.
Summary updated