The Architecture Dependence of Transfer Learning

33 concepts24 papers1 proposal

When we think about transfer learning, we naturally frame it around tasks. We assume that if Task A (e.g., Natural Language Inference) is semantically close to Task B (e.g., Text Classification), then a model trained on Task A will inherently perform better on Task B. We treat "transferability" as a static property shared between two datasets.

However, a growing body of evidence reveals a fundamental flaw in this assumption: transferability is not just a property of the data, but an emergent property of the interaction between the task and the specific neural network architecture. What transfers beautifully in one model family might completely fail or cause negative transfer in another. This phenomenon forces us to move away from universal transfer learning principles and instead study how architectural biases dictate the flow of knowledge.

Why Task Similarity Metrics Fail Across Models

To avoid the massive computational cost of brute-force trial and error, researchers often use "cheap" heuristics or analytical metrics to predict which source tasks will transfer well to a target task. For instance, one might compute the Fisher Information Matrix, analyze internal activations, or look at gradient alignments across tasks.

The catch is that these metrics are deeply coupled to the geometry of the specific model's parameter space. As Poth et al. demonstrated, a metric that accurately predicts transferability for one architecture can completely fail for another. This occurs because different architectures construct highly disparate, high-dimensional embedding spaces; a task representation that is tightly clustered and usable in one model might be scattered and uninformative in another.

[Task A Data] ---> (Architecture 1: CNN) -------> Successful Transfer
               ---> (Architecture 2: ViT) -------> Scattered Space / Failure

Similarly, Ma et al. found that task similarity measures derived from internal activations or attention head gradients are strictly bound to the specific architecture and parameter initialization from which they were extracted. When selecting optimal source data for financial time series, Long et al. likewise noted that the ideal similarity metric changes depending on whether the downstream model is a standard deep neural network or an LSTM. Even in medical imaging, Tan et al. highlighted that frameworks predicting task transferability lose their reliability when shifting from convolutional networks to vision transformers, as the underlying feature extraction mechanisms change fundamentally.

The Mismatch of Internal Representations and Biases

Why does the architecture exert such a strong veto over transfer learning? The root cause lies in how different network structures shape internal probability distributions and inductive biases.

Consider the challenge of distilling reasoning capabilities from a large language model to a smaller one. We might expect that providing the smaller model with explicit "reasoning traces" (step-by-step logic) would universally improve its performance. However, Lee et al. discovered that these traces are highly architecture-dependent. A sequence of logic that guides one model can be entirely detrimental to another if it does not align with the target model's internal probability distribution. The student model literally cannot represent the teacher's reasoning path if their architectural biases diverge.

This architectural sensitivity extends to continuous prompt transfer. While it is possible to map soft prompts into a relative space to transfer task semantics between models, Wu et al. showed that this process is highly sensitive to the source model's specific architecture and training objectives. If the source model's structural bias fails to capture the exact semantic alignment required by the target, the transfer fails completely. As Albalak et al. concluded in their dialogue benchmark study, transfer learning gains are not universal properties of the tasks themselves; instead, they emerge dynamically from the interplay between the task's objectives and the model's unique architectural biases.

The Instability of Intermediate Training

A common paradigm in deep learning is multi-stage fine-tuning: pre-train a model, fine-tune it on an intermediate "bridge" task, and finally deploy it on the target task. Yet, the benefits of this intermediate step are notoriously unstable across architectures.

Choshen et al. demonstrated that the rankings of "good" intermediate models do not exhibit architectural invariance; a winning starting point for one neural architecture might be a mediocre one for another. This variability is echoed by Zhang et al., who found that intermediate training on question-answering tasks can cause performance drops in some language model architectures while providing gains in others, depending heavily on how their internal knowledge representation dynamics operate.

This lack of invariance even applies to cross-domain reprogramming. When attempting to repurpose English language models for protein sequence representation, Chen et al. observed that the success of the cross-domain mapping is highly volatile and dependent on the underlying source model architecture. Even structural modifications designed to improve the base model itself rarely generalize; Narang et al. conducted a massive study showing that most proposed transformer modifications fail to maintain their performance gains when moved across different implementations and applications.

Downstream Bottlenecks

The architecture-dependent nature of transferability creates practical bottlenecks across various machine learning workflows:

  • Data Distillation: When using an auxiliary network to identify the most informative images for labeling (e.g., for traffic sign detection), Popov et al. noted that the complexity predictor's effectiveness is constrained by its architectural alignment with the downstream detector.
  • Domain Adaptation: In 3D object detection across different sensor platforms, Feng et al. showed that adaptation techniques that work beautifully for voxel-based or hybrid detectors can degrade performance when applied to pure point-based architectures.
  • Continual Learning: When evaluating sequential learning algorithms, Wang et al. demonstrated that the interaction between pre-trained feature representations and continual learning algorithms varies so significantly across architectures that findings based on CNNs completely fail to generalize to vision transformers.

Ultimately, these insights force a paradigm shift. We can no longer treat neural network architectures as neutral vessels that merely absorb task knowledge. Instead, the architecture dictates the very structure of what can be learned, retained, and transferred.

Key Foundational Work

A Comprehensive Study of ImageNet Pre-Training for Historical Document Image Analysis (2019) — contributed 2 concept elements to this direction.
Data Distillation for Traffic Sign Detection (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_architecture_dependent_transferability_4853c8af,
  title = {architecture-dependent transferability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/architecture-dependent-transferability/rdr_4853c8af247140d38b1e269063142996},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_architecture_dependent_transferability_4853c8af_v1,
  title = {architecture-dependent transferability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/architecture-dependent-transferability/rdr_4853c8af247140d38b1e269063142996/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake