Abstract
We aim to understand the value of additional labeled or unlabeled target data in transfer learning, for any given amount of source data; this is motivated by practical questions around minimizing sampling costs, whereby, target data is usually harder or costlier to acquire than source data, but can yield better accuracy. To this aim, we establish the first minimax-rates in terms of both source and target sample sizes, and show that performance limits are captured by new notions of discrepancy between source and target, which we refer to as transfer exponents.
Introduction
In practical machine learning, data acquisition costs vary significantly. "Source" data, such as simulations or general web crawls, is often abundant and inexpensive. In contrast, "target" data—specific to the task at hand, like medical diagnoses—is typically scarce and costly. Transfer learning seeks to leverage cheap source data to improve performance on the expensive target task. While empirical methods for transfer learning are widespread, a rigorous theoretical understanding of how to optimally combine $n_P$ source samples with $n_Q$ target samples has been lacking. This paper addresses this gap by establishing fundamental performance limits and identifying the "value" of each target label.
The Transfer Exponent
A central contribution of this work is the introduction of Transfer Exponents ($\rho$ and $\gamma$). Unlike traditional symmetric distance measures (e.g., MMD), these exponents capture the asymmetric relationship between distributions: how well the error on the source distribution controls the error on the target distribution.
As shown in Figure 3, the density of the source distribution $f_P$ relative to the target $f_Q$ near the decision boundary $h^*$ determines the rate of transfer. If the source distribution is more concentrated around the boundary ($\rho < 1$), "super-transfer" can occur, where source data is more informative than target data itself.
Algorithmic Framework
The authors propose two primary algorithmic strategies to achieve optimal transfer:
- Constrained Optimization: This approach minimizes source error subject to a performance constraint on the target data, effectively performing hypothesis transfer.
- Adaptive Sampling: This strategy utilizes unlabeled target data to decide when to "buy" expensive target labels. By sensing the difficulty of the task (the transfer exponent) on the fly, the algorithm can reach accuracy goals with minimal cost.
Figure 1 illustrates the geometric relationship between the source $P$ and target $Q$ supports. The theory remains robust even when these supports do not fully overlap, a significant advancement over previous "density ratio" theories.
Key Results
The research establishes the first minimax rates for transfer learning with two sample sizes. The error rate is shown to scale approximately as $\min(n_P^{-1/\rho}, n_Q^{-1})$.
A surprising finding is the "Oracle Simplicity": minimax optimal performance can often be achieved by a simple meta-algorithm that selects the better of two models—one trained on source data and one on target data—rather than requiring complex reweighting schemes. Furthermore, the work proves that unlabeled target data is sufficient to optimally reweight source data or select the best source, reaching near-optimal performance without needing a massive amount of expensive target labels.
Limitations and Future Work
The current analysis primarily operates under the "Relaxed Covariate Shift" assumption, where the optimal classifier for the source is also optimal for the target. While the authors explore deviations from this, the tightest results depend on this alignment. Additionally, the proposed algorithms are based on Empirical Risk Minimization (ERM), which can be computationally intensive for complex models like deep neural networks. Future research aims to extend these minimax rates to non-parametric settings where model complexity scales with data size.
Summary updated