Contrastive learning trains an encoder so that two views of the same input land close together and views of different inputs land far apart. In its most common form, called InfoNCE, the loss compares cosine similarities that have been divided by a "temperature" $\tau$, and the result depends heavily on that one number. The same InfoNCE loss is also a known lower bound on mutual information (MI), meaning how many nats of information the two views share. A separate literature has spent years on why estimators of this kind are either biased or unstable, and on how clipping the estimator's outputs trades one failure for the other. This note argues that these two problems are the same problem. When the similarity score is a cosine bounded in $[-1,1]$, dividing by $\tau$ confines the critic's log density ratio to a window of width $2/\tau$. That is the clipping device of the SMILE estimator, with the clip set by the temperature. So temperature sensitivity in contrastive learning is a bias-variance trade-off in a clipped MI estimator, and it can be controlled with a quantity that does not require knowing the true MI.
The practical payoff is a rule for choosing $\tau$. The rule uses a dimensionless signal: how close the running InfoNCE value sits to the ceiling that the current temperature and batch size impose. We will test whether a single target value of that signal transfers across batch sizes and datasets in a way that raw $\tau$ does not. If it does, the per-dataset temperature sweep is replaced by one feedback rule grounded in estimator theory. If it does not, the identity still yields a diagnostic for when a contrastive run is limited by temperature, by batch size, or by neither.
Core Claim
Let $s_{ij} \in [-1,1]$ be the cosine similarity between anchor $i$ and candidate $j$ in a batch of $K$, with $j=i$ the positive. The per-anchor InfoNCE term is $\ell_i = s_{ii}/\tau - \log\big(\tfrac{1}{K}\sum_j e^{s_{ij}/\tau}\big)$. It is largest when the positive has similarity 1 and every negative has similarity $-1$. That gives a ceiling
$$ C(\tau, K) = \log K - \log\!\big(1 + (K-1)\,e^{-2/\tau}\big) \;\approx\; \min\!\big(\log K,\; 2/\tau\big), $$
where the approximation holds because the second log term is near zero when $(K-1)e^{-2/\tau}\ll 1$ and near $\log K - 2/\tau$ when it is large. The familiar $\log K$ ceiling, proved in On Variational Bounds of Mutual Information and generalized to any distribution-free bound in Formal Limitations on the Measurement of Mutual Information, is only half the story. At the common setting $\tau = 0.5$, the temperature ceiling of about 4 nats is the tighter of the two for any $K$ above roughly 55. In those runs the bias comes from the temperature, not the batch.
The mechanism follows. Understanding the Limitations of Variational Mutual Information Estimators shows two things. First, unclipped estimators have variance that grows like $e^{I}$, where $I$ is the true MI. Second, clipping density ratios to $[e^{-c}, e^{c}]$ cuts variance by orders of magnitude at the cost of a bias that appears once the true MI exceeds what the clip allows. With a cosine critic, $c$ is $1/\tau$ around a shared offset. Low temperature is therefore a loose clip: high variance and sharp curvature, consistent with the Hessian scaling linearly in $1/\tau$ shown in Asymptotic and Finite-Time Guarantees for Langevin-Based Temperature Annealing in InfoNCE. High temperature is a tight clip: low variance and a low ceiling, consistent with the low-dimensional, redundant embeddings reported in Effective post-training embedding compression via temperature control in contrastive training. We claim that the useful operating point is best described by the saturation ratio $r = \hat I / C(\tau,K)$, where $\hat I$ is the running InfoNCE estimate, and not by $\tau$ itself.
What Is New Relative to Prior Work
The work closest to ours is the bias-variance literature on MI bounds. The $I_\alpha$ bound of On Variational Bounds of Mutual Information interpolates between the unbiased but noisy NWJ estimator and the biased but stable InfoNCE through a fixed $\alpha$, with an unscaled critic. SMILE in Understanding the Limitations of Variational Mutual Information Estimators uses a fixed clip. Both papers motivate this proposal and supply its method. Neither treats the contrastive temperature as the knob, and both leave its setting to prior knowledge of the true MI. We reuse their analysis and replace the unknown true MI with a ratio computed from the current batch.
A second group attacks the ceiling directly. FLO in Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization learns a normalizer so that tightness no longer depends on $K$. DEMI in Decomposed Mutual Information Estimation for Contrastive Representation Learning splits MI across subviews to reach $2\log K$. Neither analyzes the temperature. Our identity predicts that a cosine-critic version of either method keeps a $2/\tau$ ceiling, which may explain gains that are smaller than the removed $\log K$ ceiling would suggest.
A third group adapts temperature heuristically. Model-Aware Contrastive Learning: Towards Escaping the Dilemmas (MACL) raises $\tau$ as positive-pair alignment grows. Temperature as Uncertainty in Contrastive Learning learns a per-input $\tau(x)$ for confidence. The Langevin analysis of the annealing paper gives a schedule whose critical rate cannot be computed in practice. MACL is our main competitor. It differs from our rule in the control signal: geometric alignment for MACL, saturation against an information ceiling for us. That difference can be isolated experimentally.
One paper constrains the proposal and has to be addressed directly. Contrastive Predictive Coding Done Right for Mutual Information Estimation removes InfoNCE's calibration bias with an anchor class, yet obtains 65.74% on CIFAR-100 against 65.98% for plain InfoNCE. A naive reading says bias does not matter for representations, which would undercut any bias-variance account of temperature. Our account predicts the null result instead. The anchor corrects the $\log K$-type calibration but leaves the $2/\tau$ clip in place, so if the temperature was the binding ceiling, nothing about the representation should change. We test this directly.
Research Program
The work has three parts.
Part 1: verify the identity on data with known MI. On correlated Gaussians and the cubic transform of them, we train cosine-critic InfoNCE across a grid of $\tau$ and $K$. Separately, we train an unnormalized-critic SMILE with clip $c = 1/\tau$ and InfoNCE with its logits explicitly clipped to the same window. We then compare bias, variance, and mean squared error as true MI rises. The identity predicts three things. Cosine InfoNCE should track the clipped-logit InfoNCE closely. Its estimate should saturate at $C(\tau,K)$ rather than at $\log K$. Its variance should rise roughly exponentially in $1/\tau$ until the clip binds.
Part 2: test whether temperature sensitivity in representation learning is organized by the saturation ratio. We train SimCLR-style encoders on CIFAR-10, CIFAR-100 and ImageNet-100 over a grid of $\tau$ and batch sizes. We log $r$, the across-batch variance of $\hat I$, and the gradient norm, and record linear-probe accuracy. The claim fails if the accuracy-versus-$\tau$ curves for different $K$ and datasets do not collapse when re-plotted against the late-training value of $r$.
Part 3: build the controller, but only if Part 2 shows collapse. The controller is a slow integral update on $\log\tau$, computed from an exponential moving average of $\hat I$, that holds $r$ at a target $r^\star$. It has one hyperparameter, $r^\star$, which we set once on CIFAR-10 at one batch size and then freeze for every other dataset, batch size and architecture. We pair it with the anchor-class and FLO objectives to test whether lifting the temperature ceiling and the batch ceiling together does what lifting either alone does not.
Baselines
The naive reference point is SimCLR with the standard fixed temperature and no tuning, run on identical augmentations, encoder, epochs and batch sizes. It shows how much any temperature method is worth relative to doing nothing.
The strongest practical competitor is a per-setting grid search over fixed $\tau$. It is the most likely approach to match us, because the best fixed temperature, chosen with hindsight, is an upper bound on what any static rule can achieve. We give it a budget of roughly seven temperatures per (dataset, $K$) cell, selected on a held-out split, and compare at equal total compute by also counting our single run. Our method beats it only if it comes within about half a point of the oracle-selected accuracy while using one run per cell. As a reference point for this regime, the file reports SimCLR at 89.71% linear-probe accuracy on CIFAR-10 in the setup of $f$-Mutual Information Contrastive Learning, and 65.98% for InfoNCE on CIFAR-100 in Contrastive Predictive Coding Done Right for Mutual Information Estimation. Because training setups differ, we reproduce these numbers in our own pipeline rather than compare against them directly.
The strongest existing adaptive method is MACL, run with its published alignment-adaptive temperature and gradient reweighting, its own $\alpha$ and $A_0$ tuned with the same budget we use for $r^\star$, and then frozen across settings in the same way. Its reported ImageNet-1K figure is 66.5% top-1 at batch 1024, against 66.6% for SimCLR at batch 8192. For ImageNet-100 and CIFAR, no reported MACL number was found on Lacuna, so we run it ourselves. Beating MACL means higher accuracy or lower variance across seeds when both methods' hyperparameters are frozen and moved to settings they were not tuned on.
Two ablations isolate the key ingredient. The first keeps MACL's controller structure but feeds it our saturation ratio $r$ instead of alignment. The second keeps our controller but feeds it alignment. If the alignment-fed version transfers equally well, the information-theoretic signal adds nothing, and we will say so. We also include the logarithmic schedule $\beta(t)=c\ln(t+K)$ from the Langevin paper with $c$ grid-searched, and a learnable global $\tau$ trained by backpropagation, a common practical default for which we have no citation in the file.
Why not just…?
Why not just grid-search the temperature? This is the real threat. If the best $\tau$ for one setting transfers well to the others, a one-time sweep is cheaper than a controller. The Part 2 collapse test settles it. If the optimal raw $\tau$ moves with $K$ and with the dataset while the optimal $r$ stays fixed, as the ceiling formula predicts whenever the binding term switches between $\log K$ and $2/\tau$, then sweeps do not transfer and the controller does. If both transfer equally, we report the identity and the diagnostic, and drop the claim that the controller is needed.
Why not just use SMILE-style clipping or the $I_\alpha$ bound directly? Both carry a static threshold that, according to Understanding the Limitations of Variational Mutual Information Estimators, must be matched to an unknown true MI, and SMILE still fails the additivity self-consistency check on image data. Our point is that contrastive pipelines already apply such a clip through $\tau$. Adding a second clip only stacks two thresholds. Part 1 checks whether the two thresholds are in fact redundant.
Why not just remove the ceiling with FLO, the anchor class, or RPC? Self-supervised Representation Learning with Relative Predictive Coding removes the exponentials and is stable at small batch sizes, reaching 78.40% against CPC's 77.80% on ImageNet with ResNet-152. It does so with three new hyperparameters that need re-tuning for each architecture. FLO and the anchor class remove the $\log K$ ceiling but, according to our formula, not the $2/\tau$ ceiling. The anchor class's flat CIFAR-100 result fits that reading. The decisive test is the anchor objective run at several temperatures. If accuracy improves over plain InfoNCE only at low $\tau$, where $2/\tau$ is no longer binding, the temperature account is supported. If it never improves, the temperature account does not explain why removing calibration bias leaves accuracy unchanged.
Proposed Experiments and Decisive Checks
The synthetic study uses 20-dimensional correlated Gaussians and their cubic transform, following the two earlier bias-variance papers, with true MI swept from 2 to 20 nats, $K \in \{16, 64, 256, 1024\}$, and $\tau$ from 0.05 to 1. Metrics are bias, variance and mean squared error against the analytic MI, plus the observed plateau compared with $C(\tau,K)$. Two outcomes would falsify the identity. The first is plateaus that follow $\log K$ rather than $C(\tau,K)$ at high temperature. The second is a large gap in the bias-variance curves between cosine InfoNCE and the explicitly clipped version.
The representation study uses CIFAR-10, CIFAR-100 and ImageNet-100 with ResNet-18 or ResNet-50 encoders, three seeds per cell, and linear-probe top-1 accuracy as the primary metric. Secondary metrics are the intrinsic dimensionality of the embeddings, to test the redundancy link reported for high temperatures, and the across-seed spread of accuracy as a measure of instability. A single ImageNet-1K run at batch 1024 against MACL's reported 66.5% is a stretch goal, used only if compute allows.
The check that decides whether the paper has a method or only an analysis is transfer. We set $r^\star$ on CIFAR-10 at $K=256$ and apply it unchanged to the other datasets, to $K=64$ and $K=1024$, and to the anchor-class objective. Success means staying within half a point of the oracle grid-searched temperature in every cell, with smaller seed variance than MACL under the same frozen-hyperparameter protocol. Failure in either direction is informative, because the ceiling formula says in advance which settings are temperature-bound and which are batch-bound.
References
- On Variational Bounds of Mutual Information
- Formal Limitations on the Measurement of Mutual Information
- Understanding the Limitations of Variational Mutual Information Estimators
- Asymptotic and Finite-Time Guarantees for Langevin-Based Temperature Annealing in InfoNCE
- Effective post-training embedding compression via temperature control in contrastive training
- Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization
- Decomposed Mutual Information Estimation for Contrastive Representation Learning
- Model-Aware Contrastive Learning: Towards Escaping the Dilemmas
- Temperature as Uncertainty in Contrastive Learning
- Contrastive Predictive Coding Done Right for Mutual Information Estimation
- $f$-Mutual Information Contrastive Learning
- Self-supervised Representation Learning with Relative Predictive Coding