Transfer learning has historically been a story of classification. We pre-train on ImageNet (discrete categories) and fine-tune on medical images (discrete diagnoses). However, many of the most critical problems in science and engineering—such as predicting molecular properties, coordinate tracking, or physical constants—require predicting continuous values.
The transition from discrete labels to continuous targets breaks many of our standard transfer learning tools. Recent research has focused on bridging this gap by rethinking how we estimate "transferability" and how we align latent representations when there are no clear class boundaries.
The Failure of Class Separability
In classification, we measure the quality of a pre-trained model by how well it separates data into distinct clusters. Metrics like the Gaussian Bhattacharyya Coefficient (GBC), explored by Michal Pándy and Thomas Mensink in Transferability Estimation using Bhattacharyya Class Separability, rely on the idea that a good model should minimize the overlap between probability distributions of different classes.
When labels are continuous, there are no "classes" to separate. To use these classical metrics, researchers often resort to discretization: chopping the continuous target range into artificial "bins" or pseudo-classes. As noted by Prabhant Singh and Joaquin Vanschoren, this process is noisy and often loses the fine-grained information essential for regression.
Efficient Transferability Estimation
If we can't rely on class separation, how do we predict which pre-trained model will work best for a new regression task without the "trial and error" of full fine-tuning?
Cuong N. Nguyen and colleagues proposed a simpler path in Simple Transferability Estimation for Regression Tasks. Instead of complex information-theoretic metrics, they introduced Linear MSE and Label MSE. The intuition is elegant: if a model's frozen features can be mapped to the target labels using a simple linear layer with low error, the model has already captured the necessary underlying structure of the data.
Other researchers have looked toward Optimal Transport to bridge the gap. Tomoyuki Hatakeyama and Toshihiko Yamasaki demonstrated that by treating the relationship between features and labels as a distribution-matching problem, one can estimate transferability across both classification and regression tasks simultaneously, providing a unified metric for model selection.
Geometric Alignment and Manifold Learning
When data is scarce—a common problem in molecular property prediction—simply selecting the right model isn't enough. We need to align the source and target knowledge more deeply.
In classification, we align clusters. In regression, we must align the "shape" of the data. Sung Moon Ko, Sumin Lee, and Sehui Han have pioneered a geometric approach to this problem. Their work on the Geometrically Aligned Transfer Encoder (GATE) treats the latent representations of a neural network as points on a curved Riemannian manifold.
The intuition behind GATE is that even if the specific values change, the local geometry of the data relationships should remain consistent. By aligning these manifolds in a "locally flat" space, the model can transfer knowledge about non-linear relationships even when the target dataset is tiny. This was further refined in the GEAR framework (Geometric Embedding Alignment via cuRvature matching), which matches the Ricci scalar curvature of the source and target spaces to ensure that the "warping" of the feature space is preserved during transfer.
Boosting the Transfer Process
While geometric methods handle the latent space, older but still powerful techniques focus on the ensemble level. David Pardoe and Peter Stone were among the first to adapt boosting to this domain in Boosting for Regression Transfer.
Their algorithms, like Two-stage TrAdaBoost.R2, work by iteratively re-weighting source data. If a piece of source data contradicts the target data, its weight is reduced. This allows the model to "filter" the source knowledge, keeping only what is relevant to the continuous target task, effectively preventing "negative transfer" where unrelated source data actually hurts performance.
Summary of the Shift
The research direction in this field is moving away from "classification-first" adaptations and toward native regression frameworks. Whether through simple linear probes (Linear MSE), distribution matching (Optimal Transport), or sophisticated manifold alignment (GATE/GEAR), the goal is to treat the continuity of the target variable as a feature to be leveraged rather than a hurdle to be discretized.