Heterogeneous Transfer Learning across Disparate Feature Spaces

This direction is merged into another direction: heterogeneous feature spaces. Its last version stays readable and citable here.

Version 1 of 152 papers55 conceptsfrozen 2026-10-04

Traditional transfer learning operates on a comfortable assumption: the "shape" of knowledge is the same in both the source and the target. If you are transferring a model from one image dataset to another, you assume both use pixels as inputs and share the same number of color channels. This is homogeneous transfer learning.

In the real world, this assumption frequently breaks. A company might want to transfer fraud detection logic from a data-rich English market to a new market using different transaction features. A hospital might want to use a model trained on high-resolution MRI scans to assist with low-resolution ultrasound data. Or, as Cun-Yuan Xing notes in One-Pass Feature Evolvable Learning, the feature space itself might evolve over time as sensors are upgraded or replaced.

Heterogeneous Transfer Learning (HTL) is the study of how to move knowledge between domains that share neither the same feature dimensions, the same data modalities, nor the same label spaces.

The Alignment Problem

The fundamental challenge of HTL is the lack of a direct correspondence. If the source domain has 100 features and the target has 50 entirely different ones, a standard neural network weight matrix simply cannot be applied.

Researchers like Peilin Zhao and Steven Hoi addressed this in their Online Transfer Learning (OTL) framework. They proposed that even when feature spaces don't match, we can maintain a "mapping" that translates the target's partial information into a format the source model understands.

The intuition is to find a shared latent subspace. Imagine two people who speak different languages but both understand mathematics. To communicate, they don't translate every word; they project their ideas into the "subspace" of equations. Similarly, Yong Luo and Dacheng Tao developed Heterogeneous Multi-Task Metric Learning (HMTML), which uses unlabeled data to find high-order correlations between disparate modalities (like images and text), projecting them into a common manifold where distances can be measured consistently.

Semantic Bridging and Metaphors

If the data itself is too different to align geometrically, we can use "side information" to bridge the gap.

In Sharing Knowledge for Meta-learning with Feature Descriptions, Atsutoshi Kumagai demonstrates that natural language can act as a universal translator. By describing features (e.g., "Average temperature" vs "Celsius reading"), a pre-trained language model like BERT can recognize that two different variables in two different datasets actually represent the same concept.

A more abstract version of this is found in the work of Omer Levy and Shaul Markovitch. In Teaching Machines to Learn by Metaphors, they argue that a machine can learn a "metaphor"—a transformation function—that maps a complex, unknown target problem back onto a well-understood source problem. This allows the model to "reuse" its existing logic by viewing the new data through the lens of the old.

Transferring without Shared Structure

Recent work has pushed HTL to the extreme, attempting to transfer knowledge between models that share nothing in common—neither architecture, task, nor modality.

Kunxi Li and colleagues introduced MergeNet, which treats the parameter space itself as the carrier of knowledge. Instead of trying to align the data (the inputs), MergeNet uses attention mechanisms to allow a target model to "query" the weights of a source model. This effectively migrates "skills" from one neural network to another, even if one was trained for computer vision and the other for natural language processing.

The utility of HTL is most visible in Bayesian Optimization (BO), where we try to find the best hyperparameters for a model. Often, we have data from past experiments, but the new experiment has a different "search space" (e.g., different tunable parameters).

Frameworks like HyperBO and the work of Aryan Deshwal in Sample-Efficient BO for Heterogeneous Search Spaces show that we can "warm-start" optimization by learning the underlying "landscape" of the problem. Even if the dimensions change, the "peaks" and "valleys" of the objective function often follow similar structural patterns that can be transferred to speed up the search for the global optimum.

Why It Works: The Latent Consistency

The reason HTL is possible at all is the Latent Consistency Hypothesis. While surface-level representations (pixels, sensors, languages) vary wildly, the underlying causal mechanisms or logical structures of the tasks often remain stable.

Whether it is AC-Norm recalibrating feature channels in medical images or LM-HTR mapping search rankings across languages, the goal is always the same: to strip away the domain-specific "noise" of the feature space to reveal the transferable "signal" of the underlying task.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_heterogeneous_transfer_learning_across_d_cce05ba0,
  title = {Heterogeneous Transfer Learning across Disparate Feature Spaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heterogeneous-transfer-learning-across-disparate-feature-spaces/rdr_cce05ba0dfca4b1aadc4be2491428adf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_heterogeneous_transfer_learning_across_d_cce05ba0_v1,
  title = {Heterogeneous Transfer Learning across Disparate Feature Spaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heterogeneous-transfer-learning-across-disparate-feature-spaces/rdr_cce05ba0dfca4b1aadc4be2491428adf/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}