Abstract
Training deep neural networks on small datasets often leads to overfitting or numerical instability. This paper introduces a "mentored learning" framework where a large, pre-trained "mentor" network provides layer-by-layer guidance to a smaller "mentee" network. This approach allows the smaller model to achieve better generalization and stability than if it were trained independently, effectively bridging the gap between massive pre-trained models and small-scale data constraints.
Introduction
In modern computer vision, the standard practice is to fine-tune massive, pre-trained models like VGG-19. However, these models are often too computationally expensive for portable devices. Conversely, training mid-sized networks from scratch on small datasets is challenging due to noisy gradients and potential weight explosion. The authors propose "Generalized Mentored Learning" to address this, leveraging a large model's internal representations to stabilize and improve the training of a smaller counterpart.
Methodology
The core of the proposed framework is the use of intermediate representations from a mentor network to guide a mentee.
Layer-wise Probes and Regularization
Unlike traditional knowledge distillation that focuses solely on the final softmax output, this method defines "probes" between specific layers of the mentor and the mentee. An RMSE (Root Mean Square Error) loss is calculated between the activations of these layers, acting as a guided regularizer. This forces the mentee’s internal features to resemble the mentor’s proven features, preventing weight instability.
Mentee Personalities
The balance between mentor guidance and task labels is controlled via annealing functions ($\alpha, \beta, \gamma$). The authors define two primary "personalities":
- Obedient Mentee: Prioritizes mentor representations during an initial "mentoring phase" before shifting focus to task labels in a "self-study phase."
- Adamant Mentee: Focuses on labels almost immediately, using the mentor only as a light guide throughout training.
Additionally, Unsupervised Mentoring allows a mentee to learn features from a mentor using unlabeled data, serving as a sophisticated weight initialization strategy.
Experimental Results
The framework was evaluated on standard benchmarks including MNIST, CIFAR-10/100, and Caltech-101/256.
Performance Gains
Mentee networks consistently outperformed independent baselines:
- Low-Data MNIST: With only 1 sample per class, the Mentee achieved 48.5% accuracy (vs. 41.5% independent). Using unsupervised mentoring as an initializer, accuracy reached 96.7%.
- Caltech-101: A mentee mimicking VGG-19 reached 56.16% accuracy, significantly higher than the 45.46% baseline.
- Generalization: Features learned via mentoring on Caltech-101 proved more robust when fine-tuned on Caltech-256 (66.12% vs. 55.45%).
Key Insights
- Numerical Stability: Mentoring acts as a "safety rail," allowing for much higher learning rates (up to 0.5) without gradient explosion.
- Representation Matters: Guiding intermediate layers is crucial for building robust, portable models, moving beyond simple softmax distillation.
- Data Scarcity: The benefits of mentoring are most pronounced when labeled data is scarce. With ample supervision, the mentor's guidance can become redundant.
Conclusion
Mentored learning provides a powerful tool for training efficient models on small datasets. By leveraging the internal wisdom of large networks, mentees can achieve high performance and stability. Future work includes exploring more complex personalities and automated layer selection for mentoring.
Summary updated