Omni-Training for Data-Efficient Deep Learning

Yang Shu, Zhangjie Cao, Jinghan Gao, Jianmin Wang, Mingsheng LongarXiv 2021 · OpenReview

Abstract

Omni-Training is a novel framework designed to bridge the gap between pre-training and meta-training, addressing the "transferability dilemma" inherent in few-shot learning. By employing a "tri-flow" architecture and a self-distillation loss, the framework simultaneously optimizes for domain and task transferability. Experimental results across classification, regression, and reinforcement learning demonstrate that Omni-Training significantly outperforms state-of-the-art methods, particularly in challenging cross-domain scenarios.

1. Introduction: The Transferability Dilemma

Few-shot learning (FSL) seeks to enable models to generalize to new tasks with minimal examples. Currently, two paradigms dominate: Pre-training, which excels at handling domain shift (distributional changes in data) by learning robust features on large datasets, and Meta-training, which excels at handling task shift (adapting to new semantic categories) through "learning to learn."

Conceptual overview of Domain Shift vs. Task Shift and the proposed focus on dual transferability.

However, real-world applications often involve both shifts simultaneously. The authors identify a "transferability dilemma": pre-training is superior for domain transfer but lacks task-specific agility, while meta-training is superior for task transfer but remains brittle under domain distribution changes. Simple ensembles or joint training strategies fail to effectively combine these strengths.

2. Methodology

Omni-Training introduces two core innovations to resolve this dilemma: the Omni-Net architecture and the Omni-Loss objective.

2.1 Omni-Net (Tri-Flow Architecture)

Omni-Net replaces standard backbones with three interdependent data flows:

  • Joint-Flow: A shared layer that extracts common features and coordinates information across the network.
  • Pre-Flow: A parallel branch dedicated to preserving domain-robust features learned via pre-training.
  • Meta-Flow: A parallel branch optimized for rapid task adaptation through meta-learning.

This "tri-flow" design allows the model to maintain specialized knowledge while enabling cross-pollination of features through the shared joint-flow.

2.2 Omni-Loss and Self-Distillation

The training objective integrates both pre-training and meta-training losses. To stabilize learning and enhance transferability, the authors implement a self-distillation strategy. An Exponential Moving Average (EMA) of the model parameters serves as a "temporal ensemble" teacher, regularizing the parallel flows and ensuring that the learned representations remain robust throughout the training process.

3. Experimental Evaluation

The framework was evaluated on diverse benchmarks including classification (mini-ImageNet, CUB, and multi-domain suites), sinusoid regression, and reinforcement learning (2D Navigation and Locomotion).

Empirical evidence of the transferability dilemma and the performance of baseline combination strategies.

3.1 Key Results

  • Cross-Domain Performance: In the mini-ImageNet → CUB benchmark, Omni-Training achieved 75.83% accuracy in 5-shot tasks, significantly outperforming standard GNN (65.56%) and baseline pre-training (65.57%).
  • Regression Efficiency: In sinusoid regression, Omni-Training achieved lower Mean Squared Error (MSE) than MAML, particularly when adaptation steps were limited.
  • Reinforcement Learning: In 2D Navigation, the model reached higher expected returns faster than both MAML and pre-training baselines.

4. Discussion and Conclusion

Omni-Training effectively addresses the trade-off between domain and task transferability. Attention map visualizations reveal that while pre-training focuses on broad context and meta-training focuses on discriminative details, Omni-Training balances these perspectives to focus on relevant objects while ignoring background noise.

While the tri-flow architecture adds some implementation complexity and introduces new hyperparameters, its parameter efficiency and superior performance make it a compelling choice for data-efficient deep learning. Future research will explore scaling this approach to larger Vision Transformers and foundation models.

Summary updated

BibTeX
@misc{shu2021omni,
  author = {Yang Shu and Zhangjie Cao and Jinghan Gao and Jianmin Wang and Mingsheng Long},
  title = {{Omni-Training for Data-Efficient Deep Learning}},
  year = {2021},
  url = {https://openreview.net/forum?id=rz-5IMlk1i},
}
Figure