Sufficiency-principled Transfer Learning via Model Averaging

Xiyuan Zhang, Huihang Liu, Xinyu ZhangarXiv 2025 · arXiv

Abstract

When the transferable set is unknowable, transferring informative knowledge as much as possible—a principle we refer to as sufficiency—becomes crucial for enhancing transfer learning effectiveness. However, existing transfer learning methods not only overlook the sufficiency principle, but also rely on restrictive single-similarity assumptions (e.g., individual or combinatorial similarity), leading to suboptimal performance. To address these limitations, we propose a sufficiency-principled transfer learning framework via unified model averaging algorithms, accommodating both individual and combinatorial similarities. Theoretically, we establish the asymptotic/high-probability optimality, enhanced convergence rate and asymptotic normality for multi-source linear regression models with a diverging number of parameters, achieving sufficiency, robustness to negative transfer, privacy protection and feasible statistical inference. Extensive simulations and an empirical data analysis of Beijing housing rental data demonstrate the promising superiority of our framework over conventional alternatives.

Introduction

Transfer learning aims to improve a model's performance on a "target" task by leveraging data from related "source" domains. However, two major hurdles often lead to suboptimal results: the unknown nature of the transferable set and restrictive similarity assumptions. In practice, researchers rarely know which sources are truly helpful. Including non-informative sources can lead to negative transfer, where auxiliary data actually degrades the target model's performance. Furthermore, most existing methods assume either individual similarity (a single source is close to the target) or combinatorial similarity (a weighted mix of sources approximates the target). Real-world data rarely fits neatly into just one of these categories.

Illustration of the sufficiency principle and nested candidate domain construction. Informative sources (blue) are prioritized to achieve a sufficient model set (M2) while excluding non-informative sources (red).

The authors propose the principle of sufficiency: transferring as much informative knowledge as possible. This framework is designed to be "blind" to the transferable set while achieving optimal performance and allowing for valid statistical inference, such as calculating confidence intervals.

Methods

The proposed framework utilizes a unified model-averaging approach with three primary algorithms:

  • Trans-MAI (Individual Similarity): Designed for cases where specific sources are highly similar to the target. It employs a novel weight selection criterion with a sufficiency penalty that adaptively favors domains with larger sample sizes or better coverage of the target distribution.
  • Trans-MAC/Trans-MACs (Combinatorial Similarity): These extend the framework to scenarios where no single source is a perfect match, but a combination of sources provides a good approximation.
  • Regression Cube Technique: To address data privacy, the framework requires only summary statistics (Gram matrices and coefficient vectors) rather than raw data, enabling distributed computation while protecting individual-level information.

The core innovation is the weight selection criterion, which balances predictive performance with domain sufficiency. Unlike standard model averaging, this approach can asymptotically assign zero weight to non-informative domains, effectively filtering the sources.

Experimental Evaluation

The framework was evaluated through extensive simulations and real-world data analysis:

  1. Simulations: Four experimental setups covered homogenous and heterogeneous designs, varying the number of informative sources, sample sizes, and the presence of combinatorial similarity.
  2. Beijing Housing Rental Data: This dataset consists of 2,165 observations across four districts (Chaoyang, Shijingshan, Fengtai, and Xicheng). The goal was to predict monthly rental prices by treating different districts as sources for one another.

Key Results

  • Superior Accuracy: In simulations, Trans-MAI and Trans-MAC consistently outperformed conventional alternatives like OLS (Ordinary Least Squares), Trans-Lasso, and standard Model Averaging (MAP), particularly as the number of informative sources increased.
  • Negative Transfer Mitigation: While simple data pooling (OLS-Pool) performed poorly due to negative transfer from irrelevant sources, the proposed framework maintained high performance by effectively "zeroing out" non-informative sources.
  • Real-World Performance: In the Beijing housing analysis, Trans-MAC achieved the lowest median scaled Mean Squared Prediction Error (MSPE) across almost all districts, demonstrating more stable and accurate predictive performance than state-of-the-art transfer learning methods.

Conclusion and Future Work

This work introduces a robust framework for multi-source transfer learning that maximizes informative knowledge transfer while mitigating negative transfer. By proving asymptotic normality for their estimators, the authors enable reliable statistical inference even without prior knowledge of the transferable set.

Future research directions include extending the framework to non-linear and non-parametric settings, developing a one-step weight selection criterion for combined similarity types, and allowing for multiple candidate models per source domain to increase flexibility for complex datasets.

Summary updated

Abstract

When the transferable set is unknowable, transfering informative knowledge as much as possible\textemdash a principle we refer to as \emph{sufficiency}, becomes crucial for enhancing transfer learning effectiveness. However, existing transfer learning methods not only overlook the sufficiency principle, but also rely on restrictive single-similarity assumptions (\eg individual or combinatorial similarity), leading to suboptimal performance. To address these limitations, we propose a sufficiency-principled transfer learning framework via unified model averaging algorithms, accommodating both individual and combinatorial similarities. Theoretically, we establish the asymptotic/high-probability optimality, enhanced convergence rate and asymptotic normality for multi-source linear regression models with a diverging number of parameters, achieving sufficiency, robustness to negative transfer, privacy protection and feasible statistical inference. Extensive simulations and an empirical data analysis of Beijing housing rental data demonstrate the promising superiority of our framework over conventional alternatives.
BibTeX
@misc{zhang2025sufficiency,
  author = {Xiyuan Zhang and Huihang Liu and Xinyu Zhang},
  title = {{Sufficiency-principled Transfer Learning via Model Averaging}},
  year = {2025},
  eprint = {2507.15416},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.15416},
  howpublished = {arXiv},
}
Figure