Transfer Learning through Enhanced Sufficient Representation: Enriching Source Domain Knowledge with Target Data

Huang, Jian, Yeheng Ge, Xueyu Zhou, Jian HuangarXiv 2025 · DOI · arXiv

Transfer learning (TL) has become a cornerstone of modern machine learning, particularly when target tasks suffer from data scarcity. However, traditional TL methods often rely on rigid assumptions, requiring source and target domains to share similar functional forms or model parameters. This brittleness limits their effectiveness in real-world scenarios where domain heterogeneity and task mismatch are common—such as using classification data to improve a regression model.

To address these challenges, we introduce Transfer Learning through Enhanced Sufficient Representation (TESR). Instead of transferring model parameters, TESR focuses on transferring data representations, enabling flexible knowledge sharing across fundamentally different tasks.

The TESR Framework

TESR operates on the principle that while models may differ, the underlying information required for a task can be captured in a Sufficient Representation. The framework follows a two-step process to build a robust feature set for the target task.

Overview of the TESR framework showing the extraction of invariant representations from source data and enhancement with target-specific components.

Step I: Learning the Sufficient and Invariant Representation (SIRep)

The first stage involves analyzing multiple source domains to identify a representation ($R_c$) that is both "sufficient" (containing all relevant information for source tasks) and "invariant" (maintaining a consistent distribution across different source environments). Unlike complex generative models, TESR utilizes Energy Distance for regularization, providing a more stable and computationally efficient training process compared to GAN-based alternatives.

Step II: Representation Enhancement and Independence

Recognizing that source information may not be perfectly sufficient for a new target task, TESR learns an additional "booster" component ($R_t$) using the limited target data. A critical innovation here is the Independence Constraint: the model forces $R_t$ to be independent of the source representation $R_c$. This ensures the model efficiently uses target data to "fill in the gaps" rather than redundantly re-learning features already captured from the source.

Geometric interpretation of representation enhancement, illustrating the transition toward an orthogonal (independent) relationship between source and target components.

Experimental Evaluation

The effectiveness of TESR was validated across diverse benchmarks, demonstrating its superiority over standard Deep Neural Networks (DNN) and state-of-the-art transfer methods like TransIRM.

  • Bioinformatics (GTEx): In predicting the expression of the JAM2 gene across 13 distinct brain tissues, TESR achieved the lowest Mean Square Error (MSE) in 10 out of 13 cases, showcasing its ability to handle high biological heterogeneity.
  • Computer Vision (PACS + CIFAR-10): Using CIFAR-10 as a source, TESR significantly improved performance on the PACS dataset (Photo, Art, Cartoon, Sketch). While it excelled in most categories, the "Sketch" domain highlighted a limitation: when target data is extremely simple, complex source features can occasionally act as noise.
  • Synthetic Simulations: In complex non-linear settings where source and target tasks had different active predictors, TESR consistently outperformed competitors, proving robust even when source data was only partially relevant.

Discussion and Future Work

TESR demonstrates that Representation > Parameters. By focusing on the information bottleneck rather than the mathematical structure of the model, we can achieve more resilient transfer learning. The framework's inherent robustness allows it to default to standard supervised learning if source data is irrelevant, preventing "negative transfer."

Future research will focus on automating hyperparameter selection for representation dimensions and exploring alternative independence measures, such as Mutual Information, to further refine the synergy between source and target knowledge.

Summary updated

BibTeX
@misc{jian2025transfer,
  author = {Huang, Jian and Yeheng Ge and Xueyu Zhou and Jian Huang},
  title = {{Transfer Learning through Enhanced Sufficient Representation: Enriching Source Domain Knowledge with Target Data}},
  year = {2025},
  eprint = {2502.20414},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.20414},
}
Figure