In deep learning, the paradigm of pretraining followed by downstream fine-tuning has revolutionized computer vision and natural language processing. Instead of forcing a model to learn the fundamental syntax of language or the basic geometry of the visual world from scratch for every new task, pretraining provides a robust representational baseline. Yet, across many highly specialized domains—ranging from single-cell genomics and molecular modeling to time series forecasting and multimodal sentiment analysis—training from scratch remains surprisingly prevalent.
When a model is trained from scratch, it must simultaneously learn low-level feature representations and high-level task logic from limited, domain-specific datasets. This often leads to optimization bottlenecks, poor generalization, and severe data inefficiency. A growing body of research highlights the massive, unrealized potential of transfer learning and unsupervised pretraining across diverse scientific and engineering disciplines.
The Limitations of Training from Scratch
Building neural networks entirely on task-specific data introduces significant constraints. For instance, in temporal language localization for videos, Xiaoye Qu et al. developed an iterative attention network in Fine-grained Iterative Attention Network for Temporal Language Localization in Videos but noted that omitting a pretraining strategy limited their data efficiency and ultimate performance. Similarly, in audio source separation, Ning Zhang et al. designed a weakly supervised adversarial framework in Weakly Supervised Audio Source Separation via Spectrum Energy Preserved Wasserstein Learning that operates without auxiliary datasets or pretrained models, identifying this omission as a primary reason for performance gaps against potential transfer-learning alternatives.
Even when models exhibit remarkable innate capabilities, the absence of training or pretraining leaves their full capacity unexplored. For example, Huanwen Chen et al. demonstrated in Role of Lateral Inhibition on Visual Number Sense that a neural network utilizing lateral inhibition can possess an innate "number sense" without any training at all. However, the authors leave open the question of how a structured training or pretraining regime could systematically enhance or adapt these hardcoded biological mechanisms. In speech and language processing, architectures like the multi-scale residual network proposed by Mijit Ablimit et al. in MSRN and Multi-Headed Attention Mechanism for Language Identification miss out on reduced training times and heightened accuracy by bypassing pretraining entirely.
Domain-Specific Barriers to Pretraining
If pretraining is so effective, why isn't it universally adopted? The answer often lies in the lack of standardized, large-scale datasets and curated model repositories tailored to specific data modalities.
Consider time series forecasting. While computer vision practitioners can easily download a model pretrained on ImageNet, the time series domain has historically lacked equivalent foundational models. As Julien Herzen et al. highlight in Darts: User-Friendly Modern Machine Learning for Time Series, the absence of a curated set of deep learning models pretrained on extensive time series datasets forces practitioners to train models from scratch for every new forecasting problem, slowing down development cycles.
A similar bottleneck occurs in biomedical and molecular applications. In single-cell RNA sequencing (scRNA-seq), deep clustering networks face severe optimization and convergence difficulties. While vast repositories of annotated single-cell data exist, Simai Zhang et al. point out in Attention-based deep clustering method for scRNA-seq cell type identification that the field has yet to fully exploit these datasets for unsupervised pretraining, leaving substantial improvements in model generalization unrealized.
In biochemistry, graph neural networks (GNNs) designed for site-of-metabolism prediction, such as the framework by Vladimir Porokhin et al. in Using graph neural networks for site-of-metabolism prediction and its applications to ranking promiscuous enzymatic products, are typically restricted to narrow datasets; the authors suggest that pretraining on massive, general molecular datasets would dramatically expand their predictive accuracy and scope. This pattern extends to neuroimaging: YUXIANG WEI et al. note in Hierarchical Spatio-Temporal State-Space Modeling for fMRI Analysis that their state-space model for fMRI analysis could achieve significantly higher efficiency and capacity if initialized via a dedicated pretraining strategy rather than trained end-to-end from scratch.
The Underutilization of Existing Models
In some domains, pretraining is used, but only superficially. Models often adopt pretrained backbones without fully exploiting their latent representational capacity or adapting them to the nuances of the target task.
In monocular depth estimation, the SuperDepth framework by Sudeep Pillai et al. in SuperDepth: Self-Supervised, Super-Resolved Monocular Depth Estimation achieves high-resolution outputs through self-supervised training from scratch, yet the authors acknowledge that integrating standard ImageNet weights remains an unexploited avenue for further performance gains. A similar phenomenon occurs in multimodal video recognition: Xiao Wang 0014 et al. utilize an ImageNet-initialized backbone in Unleashing the Power of CNN and Transformer for Balanced RGB-event Video Recognition but conclude that their framework fails to fully unleash the power of large pretrained models, calling for more advanced transfer learning techniques.
Furthermore, when models are applied to complex linguistic tasks like text-to-SQL generation, basic augmentations with models like BERT demonstrate clear benefits, as shown by Huajie Wang et al. in PG-GSQL: Pointer-Generator Network with Guide Decoding for Cross-Domain Context-Dependent Text-to-SQL Generation. However, deeply integrating these language models so they seamlessly interact with specialized components (like pointer-generator networks) remains an active challenge. In emotion classification for code-mixed text, Jinghui Zhang et al. show in Emotion classification on code-mixed text messages via soft prompt tuning that soft prompt tuning can better expose the inherent capabilities of pretrained language models, proving that the bottleneck is often not the model itself, but how effectively we prompt or fine-tune it.
Unlocking Performance via Large-Scale Pretraining
When fields successfully transition away from training from scratch, the performance gains are often predictable and profound, governed by the same scaling laws observed in language models.
The most compelling evidence of this transition comes from molecular science. For years, the potential benefits of scaling up datasets and architectures for molecular graphs remained largely unexplored. Xiaohong Ji et al. addressed this gap in Exploring Molecular Pretraining Model at Scale by introducing Uni-Mol2, a molecular pretraining model scaled to 1.1 billion parameters. The authors systematically demonstrated that molecular representation learning obeys predictable scaling laws: as model size and pretraining data scale up, downstream performance across a wide array of chemical tasks improves monotonically.
[ Traditional Approach ] --> Train from Scratch --> Limited Data, High Variance
vs.
[ Pretraining Paradigm ] --> Large Unsupervised Data --> General Representations --> Fine-tuning
Whether through unsupervised pretraining in evolutionary computation surrogates (Xianrong Zhang et al. in Island-Based Evolutionary Computation with Diverse Surrogates and Adaptive Knowledge Transfer for High-Dimensional Data-Driven Optimization), multi-task learning in sentiment analysis (Leimin Tian et al. in Polarity and Intensity: the Two Aspects of Sentiment Analysis), or masked feature generation in few-shot learning (Yunlong Yu et al. in Masked Feature Generation Network for Few-Shot Learning), the trajectory of modern machine learning is clear. Moving away from isolated, from-scratch training regimes toward large-scale, pretrained foundational representations is the key to unlocking the next generation of performance across specialized scientific and engineering domains.
Key Foundational Work
Polarity and Intensity: the Two Aspects of Sentiment Analysis (2018) — contributed 1 concept elements to this direction.
Superdepth: Self-supervised, super-resolved monocular depth estimation (2019) — contributed 1 concept elements to this direction.
Weakly Supervised Audio Source Separation via Spectrum Energy Preserved Wasserstein Learning (2018) — contributed 1 concept elements to this direction.