Neural Methods for Point-wise Dependency Estimation

Yao-Hung Hubert Tsai, Han Zhao, Makoto Yamada, Louis-Philippe Morency, Russ R. Salakhutdinov

This paper shifts the focus of dependency modeling from aggregate statistics to instance-level relationships. By introducing stable neural methods for estimating Point-wise Dependency (PD), the authors provide a more robust alternative to traditional Mutual Information (MI) estimators, with significant implications for self-supervised learning and cross-modal retrieval.

The Problem: Aggregate vs. Point-wise Dependency

In machine learning, we often want to quantify how much two variables, $X$ and $Y$, depend on each other. Traditionally, this is done using Mutual Information (MI), an aggregate statistic that measures the average dependency across all possible outcomes.

However, relying solely on MI presents two major drawbacks:

  1. Lack of Granularity: MI cannot measure how strongly a specific pair of events $(x, y)$ are related (e.g., "how strongly does this specific audio clip correlate with this specific text snippet?").
  2. Estimation Instability: Recent "neural" MI estimators (like MINE) rely on variational bounds that involve logarithms and exponentiations. In practice, these bounds are notoriously difficult to optimize, suffering from high variance and numerical instability.

To address this, the authors argue for focusing on Point-wise Dependency (PD), defined as the density ratio $p(x, y) / (p(x)p(y))$. PD measures how much more likely a specific pair is to co-occur compared to if they were entirely independent.

Proposed Methods

The paper introduces two data-driven approaches to estimate this ratio using neural networks, bypassing the unstable variational bounds entirely:

  1. Probabilistic Classifier: This method reframes dependency estimation as a binary classification problem. A neural network is trained to distinguish between "real" pairs (sampled from the joint distribution $p(x, y)$) and "fake" pairs (sampled from the product of marginals $p(x)p(y)$). By applying Bayes' Theorem to the classifier's output, the model can extract the PD using a standard, stable binary cross-entropy loss.
  2. Density-Ratio Fitting: This approach uses a least-squares objective to minimize the difference between the true PD and the model's estimate. Crucially, this objective involves no logarithms or exponentiations, making it significantly more numerically stable than traditional MI lower bounds.

The authors also demonstrate that aggregate MI can be easily recovered by taking the average of the log-PD values, effectively casting MI estimation as a sub-problem of PD estimation.

Key Results

The methods were evaluated across three distinct domains: synthetic correlated Gaussians, vision (MNIST & CIFAR-10), and audio-text cross-modal retrieval.

1. Lower Variance in MI Estimation

In synthetic benchmark tasks (Correlated Gaussians and a non-linear "Cubic" variant) where the ground-truth MI is known, the proposed methods exhibited significantly lower variance and bias than popular baselines like MINE, NWJ, and Jensen-Shannon (JS) estimators.

Comparison of MI estimation methods on Gaussian and Cubic datasets.

2. Better Self-Supervised Representations

Most self-supervised "contrastive" losses (like InfoNCE/CPC) implicitly try to estimate density ratios. The authors show that their Density-Ratio Fitting objective (D-RFC) can serve as a drop-in replacement for these losses. On CIFAR-10 and MNIST, using D-RFC for self-supervised learning led to higher accuracy in downstream linear evaluation tasks compared to the standard Contrastive Predictive Coding (CPC) baseline.

Downstream classification accuracy on MNIST and CIFAR-10.

3. Cross-Modal Success

On an audio-to-text retrieval task (matching human speech recordings with text representations using Speech2Vec and Word2Vec), the Probabilistic Classifier achieved a remarkable 96.24% top-1 accuracy. This demonstrates that PD can effectively capture complex, instance-level dependencies across entirely different data modalities.

Limitations and Future Work

While the authors provide a theoretical proof that a neural PD estimator exists and converges, it is an existence theorem rather than a constructive one—it does not prescribe the optimal architecture for a specific dataset.

Additionally, while these methods are highly stable, they still require the careful sampling of "negative" pairs from the marginal distribution. In very large-scale settings, this can become computationally expensive. Future work could explore more efficient sampling strategies or apply these PD estimators to detect adversarial samples, which typically exhibit low point-wise dependency with their assigned labels.

Summary updated

Abstract

Since its inception, the neural estimation of mutual information (MI) has demonstrated the empirical success of modeling expected dependency between high-dimensional random variables. However, MI is an aggregate statistic and cannot be used to measure point-wise dependency between different events. In this work, instead of estimating the expected dependency, we focus on estimating point-wise dependency (PD), which quantitatively measures how likely two outcomes co-occur. We show that we can naturally obtain PD when we are optimizing MI neural variational bounds. However, optimizing these bounds is challenging due to its large variance in practice. To address this issue, we develop two methods (free of optimizing MI variational bounds): Probabilistic Classifier and Density-Ratio Fitting. We demonstrate the effectiveness of our approaches in 1) MI estimation, 2) self-supervised representation learning, and 3) cross-modal retrieval task.
BibTeX
@inproceedings{tsai2020neural,
  author = {Yao-Hung Hubert Tsai and Han Zhao and Makoto Yamada and Louis-Philippe Morency and Russ R. Salakhutdinov},
  title = {{Neural Methods for Point-wise Dependency Estimation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=HEhWp3BYIlq},
}
Figure