A Correlation-Ratio Transfer Learning and Variational Stein's Paradox

Lu Lin, Weiyu Li

Abstract

A fundamental challenge in transfer learning is the reliance on the similarity between source and target models. In many real-world applications, such as evolving data streams or longitudinal medical studies, this similarity condition is often violated, leading to "negative transfer." This paper introduces a novel framework based on a linear correlation-ratio strategy that establishes a mathematical relationship between models even when they are dissimilar. By constructing a correlation-ratio transfer learning likelihood (cr-TLL), the authors demonstrate that information from unrelated source models can significantly improve target inference. This phenomenon, termed a "variational Stein’s paradox," is supported by theoretical global convergence rates and empirical results across synthetic and large-scale real-world datasets.

Introduction: Beyond the Similarity Constraint

In traditional transfer learning (TL), the prevailing assumption is that the source and target domains share a common structure, parameter space, or distribution. When this assumption holds, knowledge transfer is efficient. However, when the source and target are dissimilar, standard methods often fail. This is particularly problematic in dynamic environments where "emerging variables" appear—new factors in a target model that were absent in historical source data.

Instead of searching for natural similarity, this research proposes an artificial construction of relationships through a linear correlation-ratio matrix ($\Lambda$). This shift allows for the extraction of useful information from "simple" source models to improve "complex" target models, effectively bypassing the requirement for parameter overlap.

Methodology: The cr-TLL Framework

The core of the proposed method is the correlation-ratio transfer learning likelihood (cr-TLL). The approach follows a structured mathematical process:

  1. Establishing the Link: The authors utilize the expectation properties of random variables to show that indices of linear correlations can evaluate the average effects of regressions. This allows for a precise link between source parameters ($\beta$) and target parameters ($\gamma, \theta$).
  2. Estimation of $\Lambda$: The correlation-ratio matrix is estimated using historical data or a small subset of the current sample where "permanent variables" (variables present in both source and target) are available.
  3. The Objective Function: A joint log-likelihood function is constructed, combining the target likelihood with a weighted version of the source likelihood. This weighting is determined by the estimated dispersion of each model.
  4. Global Convergence: Unlike many TL methods that only guarantee local consistency, the authors prove a global convergence rate of $\sqrt{n}$ (where $n$ is the total sample size). Remarkably, this rate holds even if the source and target models are fundamentally unrelated.

The Variational Stein’s Paradox

A key conceptual contribution of this work is the illustration of a "variational Stein’s paradox." In classical statistics, Stein’s Paradox suggests that combining estimators for unrelated parameters can result in higher accuracy than estimating them individually. The authors extend this to transfer learning, showing that even "dissimilar" source data can enhance the convergence of target parameters when the correlation-ratio is properly accounted for. This suggests that the utility of source data is not limited by its similarity to the target, but rather by our ability to model the correlation between them.

Empirical Validation

Synthetic Simulations

The framework was tested using linear and Poisson regression models designed to violate similarity conditions (e.g., significant differences in regression coefficients).

  • Accuracy: The cr-TLL estimator consistently achieved lower Mean Square Error (MSE) than standard Maximum Likelihood Estimation (MLE) and state-of-the-art debiased transfer learning (DME).
  • Small Sample Efficiency: In scenarios with a very small target sample ($n_Q = 10$), the cr-TLL was nearly 1000x more accurate than MLE, reducing MSE from 0.3 to 0.0003.

Real-World Application: U.S. Airline Delays

The authors applied the method to the 2009 ASA Data Expo dataset, containing 123 million flight records. The experiment was designed to simulate the "emerging variable" problem:

  • Source Data (1987–2003): Contained 24 basic variables.
  • Target Data (2003 onwards): Introduced 5 new causal variables for delays (e.g., security and weather delays).

The cr-TLL method achieved a 99.20% prediction accuracy, resulting in only 81 errors compared to 9,573 errors produced by standard MLE. Furthermore, the coefficients produced by cr-TLL were physically intuitive (showing positive correlations for delay factors), whereas MLE results often produced nonsensical or counter-intuitive values.

Conclusion and Future Directions

This research provides a robust alternative to similarity-based transfer learning. By utilizing the linear correlation-ratio, practitioners can leverage a wider array of historical and heterogeneous data sources to improve model inference.

While the current framework is highly effective for Generalized Linear Models (GLMs) and partially linear models, future work will focus on extending these strategies to fully nonparametric and more complex semiparametric structures. Additionally, the authors aim to develop methods that relax the requirement for pre-estimating the $\Lambda$ matrix, further broadening the applicability of the variational Stein’s paradox in machine learning.

Summary updated

Abstract

A basic condition for efficient transfer learning is the similarity between a target model and source models. In practice, however, the similarity condition is difficult to meet or is even violated. Instead of the similarity condition, a brand-new strategy, linear correlation-ratio, is introduced in this paper to build an accurate relationship between the models. Such a correlation-ratio can be easily estimated by historical data or a part of sample. Then, a correlation-ratio transfer learning likelihood is established based on the correlation-ratio combination. On the practical side, the new framework is applied to some application scenarios, especially the areas of data streams and medical studies. Methodologically, some techniques are suggested for transferring the information from simple source models to a relatively complex target model. Theoretically, some favorable properties, including the global convergence rate, are achieved, even for the case where the source models are not similar to the target model. All in all, it can be seen from the theories and experimental results that the inference on the target model is significantly improved by the information from similar or dissimilar source models. In other words, a variational Stein's paradox is illustrated in the context of transfer learning.
BibTeX
@misc{lin2022correlation,
  author = {Lu Lin and Weiyu Li},
  title = {{A Correlation-Ratio Transfer Learning and Variational Stein's Paradox}},
  year = {2022},
  eprint = {2206.06086},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.06086},
}
Figure