Neural Methods for Point-wise Dependency Estimation

Yao-Hung Hubert Tsai, Han Zhao, Makoto Yamada, Louis-Philippe Morency, Ruslan Salakhutdinov

Abstract

Mutual Information (MI) is a fundamental measure of statistical dependency but is notoriously difficult to estimate in high-dimensional spaces due to the high variance of traditional variational bounds. This paper introduces neural methods for estimating Point-wise Dependency (PD), focusing on instance-level relationships rather than aggregate averages. We propose two novel estimators—the Probabilistic Classifier and Density-Ratio Fitting—which leverage the stability of classification and least-squares objectives. Our results show that these methods provide more reliable MI estimates and improve performance in downstream tasks such as self-supervised representation learning and cross-modal retrieval.

Introduction

Mutual Information (MI) has long served as the primary metric for quantifying the relationship between random variables. However, as an aggregate statistic, MI only describes the average dependency across distributions. It fails to capture the strength of association between specific pairs of events, such as a particular image and its corresponding caption. Furthermore, standard neural MI estimators, such as MINE or NWJ, often suffer from significant training instability and high variance because their objective functions involve logarithms and exponentials of neural network outputs.

In "Neural Methods for Point-wise Dependency Estimation," researchers propose a shift in focus toward Point-wise Dependency (PD). Defined by the density ratio $r(x, y) = p(x, y) / (p(x)p(y))$, PD provides a granular, instance-level understanding of data relationships. By bringing the flexibility of deep learning to PD estimation, the authors develop methods that are more stable, scalable, and informative than traditional aggregate estimators.

Proposed Neural Estimators

The authors introduce two primary data-driven approaches for PD estimation that leverage the representational power of neural networks:

  1. Probabilistic Classifier (PC): This method reformulates dependency estimation as a binary classification task. A neural network is trained to distinguish between "real" pairs (sampled from the joint distribution $p(x, y)$) and "fake" pairs (sampled from the product of marginals $p(x)p(y)$). Using a standard cross-entropy loss, this approach benefits from the inherent stability of classification objectives.
  2. Density-Ratio Fitting (DRF): This approach directly minimizes the least-square difference between the true and estimated PD. Critically, the objective function is "log-free," avoiding the numerical instabilities and gradient explosions associated with the logarithms and exponentiations found in variational MI bounds.
Comparison of neural MI estimators and their underlying objectives. The proposed methods (Variational MI Bounds, Probabilistic Classifier, and Density-Ratio Fitting) show improved stability and lower variance across Gaussian and non-linear tasks compared to baselines like CPC and MINE.

Benchmarking and MI Estimation

While the primary goal is PD estimation, these ratios can be averaged to compute aggregate MI. In benchmarking experiments using correlated Gaussians (both linear and non-linear "Cubic" tasks), the proposed methods demonstrated significantly lower variance and bias compared to established baselines. The stability of Density-Ratio Fitting is particularly evident in high-dependency regimes where traditional bounds often fail to converge.

Applications in Representation Learning

The utility of PD estimation extends directly to self-supervised learning (SSL). The authors developed Density-Ratio Fitting Coding (D-RFC), a training objective based on their DRF estimator. When evaluated on MNIST and CIFAR-10 datasets, D-RFC consistently outperformed the popular Contrastive Predictive Coding (CPC) baseline. This performance advantage held true across both shallow and deep neural architectures, suggesting that more stable dependency estimation leads to higher-quality learned representations.

Accuracy curves for self-supervised representation learning on MNIST and CIFAR-10. D-RFC consistently achieves higher accuracy than CPC and PCC across different model depths and datasets.

Cross-Modal Retrieval and Dataset Debugging

Beyond SSL, the Probabilistic Classifier was applied to an audio-text corpus for cross-modal retrieval. The model achieved a 96.24% top-1 accuracy in matching unsupervised word features from Word2Vec (text) and Speech2Vec (audio).

A unique advantage of PD estimation is its ability to perform "dataset debugging." Because PD operates at the instance level, it can identify adversarial or noisy samples—pairs that should be related but exhibit low point-wise dependency. This allows researchers to pinpoint mislabeled data or low-quality samples that aggregate statistics would otherwise obscure.

Conclusion

By shifting the focus from aggregate Mutual Information to instance-level Point-wise Dependency, this work provides a more stable and informative framework for understanding high-dimensional data relationships. While the performance remains dependent on neural architecture choices, the proposed methods offer a significant step forward in making dependency estimation a robust tool for deep learning. Future research may explore privacy-aware versions of these estimators to mitigate the risks of detecting sensitive attributes in anonymized datasets.

Summary updated

BibTeX
@inproceedings{tsai2020neural,
  author = {Yao-Hung Hubert Tsai and Han Zhao and Makoto Yamada and Louis-Philippe Morency and Ruslan Salakhutdinov},
  title = {{Neural Methods for Point-wise Dependency Estimation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=HEhWp3BYIlq},
}
Figure