Thermodynamic Transfer Evidence: Deciding How Much to Borrow From a Source Task

When a target task has few labels, practitioners borrow from a related, data-rich source task. The hard question is not how to borrow but whether to, and how much. Borrowing too much from a mismatched source makes the target model worse than training from scratch, a failure known as negative transfer. Borrowing too little wastes the source. Today this decision is made by validation splits that are unreliable when the target has fifty labels, or by architectural safeguards that come with guarantees only for squared loss. We propose to answer it with the quantity Bayesian statistics already designates for such decisions: the marginal likelihood. This is the probability the target data receives under a model whose prior has been shaped by the source. We estimate it with thermodynamic integration, a physics-derived method that computes such hard integrals by averaging along a path of gradually "cooled" distributions.

The core observation is this. If we control transfer strength with a single exponent $\alpha$ on the source likelihood, the resulting curve of target evidence against $\alpha$ has a derivative equal to a difference of two ordinary Monte Carlo averages. One average comes from a sampler that sees the source data only. The other comes from a sampler that sees both source and target. Running both samplers on a ladder of $\alpha$ values traces the whole curve in one sweep. From that curve we get a Bayes factor for transfer versus scratch (its sign is a direct test for negative transfer) and a transfer strength $\alpha^\star$ chosen by evidence rather than by a held-out split.

Core Claim

Let $\theta$ be the target model's parameters, $p(\theta)$ a base prior, $y_s$ the source data and $y_t$ the target data. We define a family of source-informed priors $p_\alpha(\theta) \propto p(y_s\mid\theta)^\alpha p(\theta)$, with $\alpha\in[0,1]$. Here $\alpha=0$ means no transfer and $\alpha=1$ means using the full source posterior as the target prior. The transfer evidence is $\mathcal{E}(\alpha)=\log\int p(y_t\mid\theta)\,p_\alpha(\theta)\,d\theta$. Write $Z(\alpha,\tau)=\int p(y_s\mid\theta)^\alpha p(y_t\mid\theta)^\tau p(\theta)\,d\theta$, the normalizing constant when the source is tempered by $\alpha$ and the target by $\tau$. Then $\mathcal{E}(\alpha)=\log Z(\alpha,1)-\log Z(\alpha,0)$. The standard thermodynamic identity (the derivative of a log-normalizer with respect to a tempering exponent is the expected log-likelihood under the tempered distribution) gives

$$\frac{d\mathcal{E}}{d\alpha}=\mathbb{E}_{\pi_{\alpha,1}}\!\left[\log p(y_s\mid\theta)\right]-\mathbb{E}_{\pi_{\alpha,0}}\!\left[\log p(y_s\mid\theta)\right],$$

where $\pi_{\alpha,\tau}\propto p(y_s\mid\theta)^\alpha p(y_t\mid\theta)^\tau p(\theta)$.

The right-hand side has a plain reading. Adding the target data pulls the posterior toward parameters that explain the source better or worse. If the target agrees with the source, adding it raises the source fit, and evidence rises with $\alpha$. If they conflict, the source fit drops, and borrowing more hurts. The anchor $\mathcal{E}(0)=\log p(y_t)$ is the target-only evidence. Integrating the derivative from $0$ to $\alpha$ gives the Bayes factor $\log B(\alpha)=\mathcal{E}(\alpha)-\mathcal{E}(0)$ against training from scratch. We claim three things. First, $\log B(\alpha^\star)$ predicts the realized sign of transfer on held-out target data better than validation-based rules when the target is small. Second, $\alpha^\star$ tracks the test-optimal transfer strength. Third, thermodynamic estimation is needed for both, because the one-step Gaussian (Laplace) approximation of the same curve misranks transfer decisions in the non-Gaussian settings where transfer matters.

The shape of the idea is borrowed from Bayesian clinical "power priors," which discount historical data with an exponent. What we have not found in the file is a thermodynamic estimator of the whole evidence curve, a use of its sign as a negative-transfer test, or an evaluation on machine-learning transfer benchmarks.

Relation to Prior Work

The method comes from two thermodynamic-integration papers. The power-posterior identity above is the one that Knuth et al. derive from the equivalence between evidence and a physical partition function. Application of referenced thermodynamic integration to Bayesian model selection shows that starting the path from a reference density close to the target, rather than from a broad prior, cuts the cost dramatically: 0.5% standard error with 308 draws versus about 55,000 for standard power posteriors on Radiata Pine. Our path over $\tau$ at fixed $\alpha$ is a referenced path whose reference is the tempered source posterior, not a Gaussian at the mode. This is exactly the cross-task reference that paper leaves unexplored. The difference is that our reference is not an approximation we hope is close. It is a scientifically meaningful prior, and its distance from the target is the object of study.

That paper also constrains the design. Its finding that the Laplace approximation selected the wrong 200-dimensional epidemiological model is why we treat Laplace as a baseline to beat, not a shortcut. The replica-exchange Hamiltonian sampler of Ndiwago et al. supplies the sampling machinery for moving across tempering rungs. Placing rungs evenly in the expectation domain, as Brekelmans et al. show for variational paths, supplies our schedule. The integrand itself is an expected log-likelihood, so we can space $\alpha$ rungs by changes in $\mathbb{E}[\log p(y_s\mid\theta)]$.

The motivating gap comes from transfer learning. Characterizing and Avoiding Negative Transfer defines negative transfer as the transferred model having higher target risk than the target-only model. It fights this with an adversarially trained filter on source samples. Our Bayes factor $B<1$ is the evidence-scale counterpart of that condition. It is computed without a discriminator, but it works per task rather than per sample. Residual Feature Integration is Sufficient to Prevent Negative Transfer gives a never-worse-than-scratch guarantee, but only under squared loss. Our criterion applies equally to classification likelihoods, which is one reason we test on classification. Jaech et al. report that transfer gains are largest in the cold-start regime and shrink as target data grows. We predict that the evidence curve reproduces this pattern, with $\alpha^\star$ falling as target size rises.

The closest competitor on the transfer side is Bayesian Multitask Learning with Latent Hierarchies. It learns tree-structured task relationships under a coalescent prior, fitted by expectation-maximization and greedy clustering, not by exact evidence. The same paper notes that short trees can over-share with irrelevant tasks. We do not propose a new hierarchy. We propose an evidence computation that could audit one, starting with the single source–target edge. Boosted multi-task learning and YieldNet share structure by design and never ask whether sharing is warranted. Those are the questions our Bayes factor answers.

Research Program

Full Markov chain Monte Carlo over deep networks is out of reach, and we will not pretend otherwise. The program therefore works at the scale where exact evidence is feasible and where transfer decisions are actually made. That means Bayesian generalized linear models, and Bayesian prediction heads on top of a fixed, shared feature extractor. We will first build the two-chain estimator. It uses a ladder of $K_\alpha$ source exponents. At each rung, replica-exchange Hamiltonian Monte Carlo draws from $\pi_{\alpha,0}$ and $\pi_{\alpha,1}$, and neighboring rungs swap states. Trapezoidal integration gives $\mathcal{E}(\alpha)$ at every rung. A single referenced path over $\tau$ at $\alpha=0$ gives the anchor $\log p(y_t)$. The cost is about $2K_\alpha$ chains. That is the same order as one ordinary thermodynamic-integration run, and it yields the entire curve rather than one number.

Second, we will analyze the estimator's error, which has two sources. Discretization error in $\alpha$ is controlled by the variance of the source log-likelihood along the path, in the same way that the variational-path analysis of Brekelmans et al. ties the gap to divergences between neighboring rungs. The second source is Monte Carlo error in the difference of two expectations. We expect this to be the dominant term when source and target conflict. In that case $\pi_{\alpha,1}$ and $\pi_{\alpha,0}$ separate sharply, and the integrand may jump, much like a phase transition. We will treat such jumps as a diagnostic in their own right: an abrupt drop in $d\mathcal{E}/d\alpha$ marks the transfer strength at which the source begins to override the target.

Third, we will extend the construction beyond one source. With $M$ candidate sources, each receives its own exponent. The gradient of the evidence with respect to each exponent is again a difference of expected source log-likelihoods. Gradient ascent on the exponent vector therefore becomes a principled way to weight sources, which we compare against discriminator gating.

Baselines

The naive reference points are target-only training ($\alpha=0$) and full pooling or plain fine-tuning from the source ($\alpha=1$). Every method must beat the better of the two on held-out target likelihood and accuracy. Without this, "transfer helps" is not established. Both are run with the identical model, prior and features as our method.

The strongest decision-rule competitor is target-side $K$-fold cross-validation over the same grid of $\alpha$ values and over the transfer/no-transfer choice. It uses the same Bayesian posterior predictive, so the only difference is how $\alpha$ is picked. It gets the same total sampler budget as our method, and fold counts are tuned per target size. We beat it if, averaged over target sizes of 100 or fewer labels, our selected $\alpha^\star$ achieves lower held-out log loss. We also require better detection of the sign of transfer, measured as area under the ROC curve against realized negative transfer.

The strongest estimator competitor is the Laplace approximation of the same curve $\mathcal{E}(\alpha)$. It uses a Hessian at each tempered mode and a matched compute budget. It isolates whether thermodynamic sampling, rather than the evidence criterion itself, is doing the work. Beating it means fewer wrong transfer decisions on the non-Gaussian classification tasks. We also require matching it on conjugate linear-Gaussian cases, where both should be exact.

The strongest learned-method competitors are REFINE (Residual Feature Integration is Sufficient to Prevent Negative Transfer) and the Discriminator Gate (Characterizing and Avoiding Negative Transfer). Both are run on the same frozen features with their published architectures and a tuning budget equal to ours. For reference, the Discriminator Gate paper reports a 7–13% average accuracy improvement over baselines on Office-Home. REFINE reports 10–15% over Adapters and LoRA on CIFAR-100→CIFAR-10. On the School benchmark, Multi-boost reports 37.7% explained variance (Boosted multi-task learning). On Amazon multi-domain sentiment, the coalescent model reports about 80.5% accuracy at 6,400 examples versus 75.8% for independent models (Bayesian Multitask Learning with Latent Hierarchies). We reproduce these setups rather than compare against copied numbers. Our heads are simpler, so absolute accuracy is not the target. The claim we test is avoidance of negative transfer at small target sizes, and near-parity elsewhere.

Two ablations isolate the key ingredient. Fixing $\alpha=1$ (the source posterior used directly as the reference, with no curve) tests whether the tempered curve matters beyond the cross-task reference. Replacing the source posterior by a Gaussian fitted at its mode tests the sensitivity of Gaussian references, which that TI paper itself flags as a risk.

Why not just…?

Why not just use the Laplace approximation of the evidence? It is cheap and would be sufficient if source-informed posteriors were near-Gaussian. Application of referenced thermodynamic integration to Bayesian model selection found it selecting the wrong model in 200 dimensions. Transfer posteriors under conflict are exactly where posteriors become skewed or bimodal, because source and target pull toward different modes. The Laplace baseline above settles this empirically. If it matches thermodynamic estimates on classification heads, we will report that the cheap estimator suffices and the contribution reduces to the criterion.

Why not just cross-validate the transfer strength on the target? With 20–100 target labels, each fold holds a handful of points, so the validation estimate is noisy. Jaech et al. locate the value of transfer precisely in this cold-start regime. The evidence uses every target label for both fitting and assessment, and it penalizes flexibility automatically. Cross-validation is a genuine threat as target size grows, and we expect it to catch up there. The crossover point is a reported result, not something to hide.

Why not just fit a hierarchical model and learn the shrinkage by empirical Bayes? This is the most serious alternative. In a Gaussian hierarchy, learning the variance tying target to source plays the same role as choosing $\alpha$. The coalescent model of Daumé III is a richer version of the same idea. It fits that variance by expectation-maximization, a point estimate that can over-share, as the paper itself notes. We will run a two-level hierarchical head with type-II maximum likelihood on the shrinkage as an extra baseline. If it matches our decisions everywhere, the honest conclusion is that exact evidence adds little over empirical Bayes. We expect a gap mainly where the source posterior is non-Gaussian and a single shrinkage variance cannot represent partial agreement.

Why not just use REFINE, which already guarantees no negative transfer? Its guarantee holds under squared loss. It also returns an architecture, not a decision or a measure of relatedness. We compare directly on classification transfer with corrupted sources. The informative outcome is whether the Bayes factor flags the corrupted sources that REFINE silently tolerates.

Proposed Experiments

We will first validate the estimator where ground truth exists. On conjugate Bayesian linear regression we compute $\mathcal{E}(\alpha)$ in closed form. On Radiata Pine we reproduce the referenced-TI error-versus-draws curve and add a source-reference variant, which is known to reach 0.5% error with 308 draws in the single-task case.

We then run leave-one-task-out transfer on the School data, with 139 schools as candidate targets and the remaining schools pooled as the source. Target sizes are subsampled down to 10 students. For classification, we use Amazon multi-domain sentiment, with logistic heads on fixed text features across all ordered domain pairs, plus the Office-Home domain pairs with Bayesian softmax heads on frozen image features. For each pair we inject controlled label corruption into the source at rates from 0% to 80%, creating a known gradient from positive to negative transfer. Corn→soybean transfer on the Corn Belt yield data, with Bayesian heads on YieldNet-style features, tests a regression pair whose relatedness is plausible but not guaranteed.

The decisive checks are three. First, the sign of $\log B(\alpha^\star)$ must predict realized negative transfer on held-out target data better than cross-validation and Laplace do, measured by ROC area across all pairs and corruption levels. Second, $\alpha^\star$ must decrease monotonically as source corruption and target size increase, and it should correlate with the test-optimal $\alpha$. Third, sharp drops in the evidence derivative must coincide with the corruption levels at which held-out accuracy collapses.

If the Bayes factor fails the first check against cross-validation below 100 target labels, the central claim is wrong. If it passes but the Laplace version passes equally well, thermodynamic integration is unnecessary and we will say so.