TL;DR
Counter-intuitively, pretraining a model on data that is dissimilar to your target can lead to significantly better forecasting performance than using the most similar data. In the volatile world of cryptocurrency, the authors show that selecting the "farthest" sources via Wasserstein distance can improve forecasting skill by over 20 percentage points for Transformers, challenging the standard "proximity-first" heuristic in transfer learning.
What Problem Does This Paper Address?
When we don't have enough data to train a model from scratch—a common issue for newly listed financial assets—we often turn to transfer learning: pretraining a model on a "source" asset before fine-tuning it on our "target."
The million-dollar question is: Which source should we pick? The conventional wisdom is to choose the most similar data. Researchers often use metrics like the Wasserstein distance to measure the "proximity" between two data distributions, assuming that the closer the source is to the target, the better the transfer.
This paper challenges that assumption. The authors argue that marginal similarity (how much the data "looks" the same) doesn't necessarily capture predictive utility. Sometimes, a source that is geometrically distant might actually be more informative because it contains a broader range of variation that helps the model learn the underlying relationships more robustly.
Methods
The researchers focused on forecasting five-day cryptocurrency volatility. Their approach involved several key components:
- Measuring Distance: They used the Gaussian-Wasserstein ($W_2$) distance to quantify the difference between the feature distributions of different crypto assets over 30-day windows.
- Source Selection Policies: They compared three main strategies for picking a pretraining source:
- Nearest $W$: Pick the asset with the smallest $W_2$ distance to the target.
- Farthest $W$: Pick the asset with the largest $W_2$ distance.
- Forward Pilot: A "brute force" validation method where they test all potential sources on a small "pilot" window of target data and pick the best performer.
- Model Architectures: They tested six different model families, ranging from simple linear models (DLinear) and recurrent networks (LSTMs, GRUs) to complex MLPs and Transformers.
- Evaluation: Performance was measured using "persistence-relative skill"—essentially, how much better the model performs compared to a simple baseline that assumes tomorrow's volatility will be the same as today's.
Datasets
The study used a "frozen panel" of daily spot USDT bars from Binance for ten major cryptocurrency assets:
- Established: BTC, ETH, BNB, XRP, SOL.
- Newer/Diverse: SEI, JTO, ONDO, SUI, TIA.
The data spanned from August 2022 to September 2026 (per the paper's specific historical panel boundaries), providing a mix of long-established histories and short, uneven tracks for newer listings.
Key Insights and Ideas
- The "Broader Variation" Advantage: The authors provide a theoretical regression example showing that if a source has a wider range of informative variation (i.e., it's "farther" away), it can estimate the shared relationship more accurately than a "near" source with cramped data.
- Learner-Specific Sensitivity: The benefit of choosing distant sources isn't universal—it depends heavily on the model. While Transformers and MLPs saw massive gains from "farthest" sources, simpler recurrent models (like GRUs) were less sensitive to the geometric distance of the source.
- Selection vs. Quality: A crucial "aha" moment is that better selection does not equal better forecasting. While Transformers benefited the most from smart source selection, they were still outperformed in absolute terms by compact recurrent models (LSTMs and GRUs).
Key Results
- Farthest Beats Nearest: For the Transformer model, selecting the three farthest sources improved skill by 20.90 percentage points over the nearest sources. For the MLP, the gain was 12.52 percentage points.
- Pilot Selection Works: Across all six model families, using a "forward pilot" to validate sources before the final test led to better mean skill than picking a source at random.
- Recurrent Models Rule: Despite the selection gains for larger models, the GRU-32 and LSTM-16 models achieved the lowest absolute Mean Squared Error (MSE), even outperforming the standard HAR (Heterogeneous Autoregressive) econometric baseline.
Limitations and Future Work
The authors are careful to note that these findings are exploratory and based on a fixed historical panel. Key limitations include:
- Fixed Budgets: The models were trained for a fixed number of epochs; different adaptation lengths might change the results.
- Computational Cost: While "geometric screening" (using $W_2$ distance to narrow down candidates) shows promise, the study didn't measure actual runtime savings.
- Generalization: The results are specific to the 10 assets and the specific time period studied.
Future work could involve testing these selection rules across different market regimes, varying the adaptation length, and exploring "multi-source" learning where models are pretrained on a diverse mix of sources rather than just one.
Summary updated