Testing sufficiency for transfer learning

Ziqian Lin, Yuan Gao, Feifei Wang, Hansheng WangComput. Stat. Data Anal 2025 · OpenReview

TL;DR

This paper introduces a novel statistical framework for testing "transfer learning sufficiency." It addresses the critical question of whether information learned from a large source dataset is truly useful for a smaller target dataset, developing a principled statistical test to determine this. This is crucial for avoiding wasted effort and ensuring that transfer learning is genuinely beneficial.

What Problem Does This Paper Address?

In many real-world scenarios, especially in fields like genomics and medical imaging, we often have a limited amount of target data for a specific task (e.g., predicting disease outcome from patient data) but access to a much larger, related source dataset (e.g., general gene expression data). Transfer learning aims to leverage the knowledge from the source data to improve model performance on the target data. However, a key challenge is knowing when this transfer is actually beneficial. Simply applying transfer learning without verification can lead to suboptimal results or even hinder performance. This paper tackles the problem of formally testing whether the information captured from the source data is sufficient to adequately represent the target data's predictive features, or if valuable predictive information remains hidden in the original target features.

Methods

The authors propose a novel testing procedure for transfer learning sufficiency. The core idea is to first develop a transfer learning estimator for the target model using a feature-generating function learned from the source data. This feature-generating function, typically a linear transformation of the original high-dimensional features, reduces the dimensionality of the data. The test then examines the relationship between the pseudo-residuals of the target model and the original features. Under the null hypothesis of transfer learning sufficiency, these pseudo-residuals should be uncorrelated with the original features. The paper develops a centralized and standardized test statistic based on the sum of squared sample covariances, rigorously deriving its asymptotic null distribution as a standard normal distribution.

Datasets

  • Dogs vs. Cats Dataset: Used as the target dataset for a binary image classification task. It contains 25,000 images.
  • ImageNet Dataset: Used as the source dataset, containing 1,281,167 images across 1000 categories.

The paper also mentions using various pretrained deep learning models (VGG, ResNet, InceptionV3) to extract high-dimensional feature vectors from these image datasets.

Key Insights and Ideas

  • Formalizing "Sufficiency": The paper moves beyond simply applying transfer learning to formally defining and testing "transfer learning sufficiency." This is crucial for understanding the theoretical underpinnings and practical applicability of transfer learning.
  • Bridging CS and Statistics: The work draws inspiration from both computer science's feature-based transfer learning approaches and statistical hypothesis testing methodologies.
  • The Power of Pseudo-Residuals: The test statistic cleverly utilizes the properties of pseudo-residuals under the null hypothesis to detect whether information has been adequately transferred.
  • High-Dimensional Challenges: The proposed method is designed to handle high-dimensional data, a common scenario in modern machine learning and statistical analysis.

Key Results

  • Theoretical Guarantees: The paper provides theoretical guarantees for the proposed transfer learning estimator and the sufficiency test, showing its asymptotic normality under the null hypothesis.
  • Simulation Performance: Extensive simulation studies demonstrate that the proposed test statistic has well-controlled Type I error rates (empirical sizes close to the nominal 5%) when the null hypothesis holds, and exhibits good power to detect transfer learning insufficiency, especially with larger signal strengths and target sample sizes.
  • Real-World Efficacy: Applying the test to a real-world image classification problem (Dogs vs. Cats) using ImageNet as the source dataset, the p-values obtained were consistently above the significance level of 0.05, supporting the null hypothesis of transfer learning sufficiency. This aligns with the high prediction accuracies achieved on the test set.

Limitations and Future Work

  • Source Data Size Requirement: The current testing procedure requires the source data sample size to be significantly larger than the target data sample size. The authors suggest investigating how to test sufficiency when these sizes are comparable.
  • Improving Estimation Post-Rejection: If the null hypothesis of sufficiency is rejected, the paper does not directly address how to improve the target model's estimation accuracy. This is identified as an important future research direction.
  • Nonlinear Feature Generation: The current framework primarily uses linear transformations for feature generation. The authors propose exploring how to test transfer learning sufficiency with nonlinear feature generating functions.

Summary updated

Abstract

Modern statistical analysis often encounters high dimensional models but with limited sample sizes. This makes it difficult to estimate high-dimensional statistical models based on target data with limited sample size. Then how to borrow information from another large sized source data for more accurate target model estimation becomes an interesting problem. This leads to the useful idea of transfer learning. Various estimation methods in this regard have been developed recently. In this work, we study transfer learning from a different perspective. Specifically, we consider here the problem of testing for transfer learning sufficiency. We denote transfer learning sufficiency to be the null hypothesis. It refers to the situation that, with the help of the source data, the useful information contained in the feature vectors of the target data can be sufficiently extracted for predicting the interested target response. Therefore, the rejection of the null hypothesis implies that information useful for prediction remains in the feature vectors of the target data and thus calls for further exploration. To this end, we develop a novel testing procedure and a centralized and standardized test statistic, whose asymptotic null distribution is analytically derived. Simulation studies are presented to demonstrate the finite sample performance of the proposed method. A deep learning related real data example is presented for illustration purpose.
BibTeX
@inproceedings{lin2025testing,
  author = {Ziqian Lin and Yuan Gao and Feifei Wang and Hansheng Wang},
  title = {{Testing sufficiency for transfer learning}},
  booktitle = {Comput. Stat. Data Anal},
  year = {2025},
  url = {https://openreview.net/forum?id=oD87cPuXw8},
}
Figure