In self-supervised learning, Mutual Information (MI) serves as the mathematical engine for learning representations. By maximizing the MI between different views of the same data, models learn to capture essential features while discarding noise. However, the most popular tool for this task—the InfoNCE loss—suffers from a fundamental statistical "ceiling": the estimate it provides is mathematically capped at the logarithm of the batch size ($\log N$).
If the true dependency between two variables is high, a contrastive estimator with a small batch size will simply stop "seeing" any further improvements, leading to sub-optimal representations. Recent research has focused on bypassing this bottleneck through better normalization, variational divergences, and direct gradient estimation.
The Origin of the Statistical Ceiling
The $\log N$ bottleneck arises because contrastive estimators like InfoNCE treat MI estimation as a multi-class classification problem where one "positive" pair must be distinguished from $N-1$ "negative" samples. As Ruizhi Liao and colleagues explain in DEMI: Discriminative Estimator of Mutual Information, this framing forces the estimator to rely on a batch-wise partition function. When the number of samples $N$ is small, the estimator lacks the "resolution" to represent high MI values, effectively saturating at $\log N$.
Interestingly, Kyungeun Lee and Wonjong Rhee demonstrate in Towards a Rigorous Analysis of Mutual Information in Contrastive Learning that this is an estimation problem, not necessarily a learning problem. Even if the training-time estimator is capped, the underlying encoder can still capture high MI. The challenge, therefore, is to develop estimators that can actually track and reward this high-information growth without requiring massive, computationally expensive batches.
Amortizing the Partition Function
One way to break the ceiling is to change how we handle the "denominator"—the partition function that normalizes density ratios. Standard contrastive methods estimate this using only the current mini-batch.
Chenyang Tao and Lawrence Carin proposed FLO (Fenchel-Legendre Optimization) in Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization. FLO introduces a "dual-critic" architecture: while one network scores pairs, a second variational function learns to approximate the normalizer itself. This "self-normalizing" behavior allows the model to estimate MI tightly without being tethered to the batch size.
A different approach to the same problem involves looking outside the current batch. In Mutual Information Estimation using LSH Sampling, Anshumali Shrivastava and Ryan Spring use Locality-Sensitive Hashing (LSH) to sample negatives from the entire dataset rather than just the mini-batch. By using LSH as a proposal distribution for importance sampling, they decouple the MI estimate from the batch size, allowing it to scale toward the log of the total dataset size.
Variational Divergences and Derangements
Another strategy involves moving away from the specific KL-divergence bounds used in InfoNCE toward more flexible $f$-divergences. Nunzio Alexandro Letizia and Nicola Novello introduced $f$-DIME in Variational f-Divergence and Derangements for Discriminative Mutual Information Estimation. They identify that standard random permutations used to create "negative" samples often include "fixed points" (where a sample is accidentally paired with itself), which introduces bias. By using data derangements—permutations specifically designed to have no fixed points—and variational $f$-divergence frameworks, they eliminate the $\log N$ cap entirely.
Similarly, Reuben Dorent and Polina Golland provide a theoretical bridge in Connecting Jensen–Shannon and Kullback–Leibler Divergences. They show that certain divergence-based bounds are inherently independent of batch size, mathematically justifying why discriminative classifiers can be more stable and accurate than traditional contrastive estimators.
Direct Estimation via Gradients and Copulas
If the goal is to train a model, we often don't need the absolute value of MI; we only need its gradient. Liangjian Wen and Mingyuan Zhou developed the Mutual Information Gradient Estimator (MIGE) in Mutual Information Gradient Estimation for Representation Learning. By using score functions of implicit distributions to estimate gradients directly, MIGE bypasses the need for negative sampling altogether, making it immune to the batch size saturation that plagues InfoNCE.
Finally, some researchers have looked toward classical statistics for inspiration. In Inductive Mutual Information Estimation: A Convex Maximum-Entropy Copula Approach, Yves-Laurent Kom Samo introduces MIND, which uses copula theory and maximum entropy. Unlike "deductive" neural estimators that try to learn complex joint distributions from scratch, MIND finds the simplest distribution consistent with the data. This inductive approach is naturally consistent even with small samples and is not bounded by the $\log N$ ceiling, providing a robust alternative for high-dimensional dependency modeling.