Abstract
Omni-Training is a novel framework designed to bridge the gap between pre-training and meta-training, addressing the "transferability dilemma" inherent in few-shot learning. By employing a "tri-flow" architecture and a self-distillation loss, the framework simultaneously optimizes for domain and task transferability. Experimental results across classification, regression, and reinforcement learning demonstrate that Omni-Training significantly outperforms state-of-the-art methods, particularly in challenging cross-domain scenarios.
1. Introduction: The Transferability Dilemma
Few-shot learning (FSL) seeks to enable models to generalize to new tasks with minimal examples. Currently, two paradigms dominate: Pre-training, which excels at handling domain shift (distributional changes in data) by learning robust features on large datasets, and Meta-training, which excels at handling task shift (adapting to new semantic categories) through "learning to learn."
However, real-world applications often involve both shifts simultaneously. The authors identify a "transferability dilemma": pre-training is superior for domain transfer but lacks task-specific agility, while meta-training is superior for task transfer but remains brittle under domain distribution changes. Simple ensembles or joint training strategies fail to effectively combine these strengths.
2. Methodology
Omni-Training introduces two core innovations to resolve this dilemma: the Omni-Net architecture and the Omni-Loss objective.
2.1 Omni-Net (Tri-Flow Architecture)
Omni-Net replaces standard backbones with three interdependent data flows:
- Joint-Flow: A shared layer that extracts common features and coordinates information across the network.
- Pre-Flow: A parallel branch dedicated to preserving domain-robust features learned via pre-training.
- Meta-Flow: A parallel branch optimized for rapid task adaptation through meta-learning.
This "tri-flow" design allows the model to maintain specialized knowledge while enabling cross-pollination of features through the shared joint-flow.
2.2 Omni-Loss and Self-Distillation
The training objective integrates both pre-training and meta-training losses. To stabilize learning and enhance transferability, the authors implement a self-distillation strategy. An Exponential Moving Average (EMA) of the model parameters serves as a "temporal ensemble" teacher, regularizing the parallel flows and ensuring that the learned representations remain robust throughout the training process.
3. Experimental Evaluation
The framework was evaluated on diverse benchmarks including classification (mini-ImageNet, CUB, and multi-domain suites), sinusoid regression, and reinforcement learning (2D Navigation and Locomotion).
3.1 Key Results
- Cross-Domain Performance: In the mini-ImageNet → CUB benchmark, Omni-Training achieved 75.83% accuracy in 5-shot tasks, significantly outperforming standard GNN (65.56%) and baseline pre-training (65.57%).
- Regression Efficiency: In sinusoid regression, Omni-Training achieved lower Mean Squared Error (MSE) than MAML, particularly when adaptation steps were limited.
- Reinforcement Learning: In 2D Navigation, the model reached higher expected returns faster than both MAML and pre-training baselines.
4. Discussion and Conclusion
Omni-Training effectively addresses the trade-off between domain and task transferability. Attention map visualizations reveal that while pre-training focuses on broad context and meta-training focuses on discriminative details, Omni-Training balances these perspectives to focus on relevant objects while ignoring background noise.
While the tri-flow architecture adds some implementation complexity and introduces new hyperparameters, its parameter efficiency and superior performance make it a compelling choice for data-efficient deep learning. Future research will explore scaling this approach to larger Vision Transformers and foundation models.
Summary updated